[OE-core][PATCH][walnascar] cve-update-nvd2-native: re-introduce CVE_SOCKET_TIMEOUT to bound urlopen()

[OE-core][PATCH][walnascar] cve-update-nvd2-native: re-introduce CVE_SOCKET_TIMEOUT to bound urlopen()

[Mehmet Fide]

From: Mehmet Fide <mehmet.fide@screeningeagle.com>

The fetch task calls urllib.request.urlopen() with no timeout argument, so
when an NVD endpoint accepts the TCP connection but stops sending data,
the call blocks forever and the existing retry loop driven by
CVE_DB_UPDATE_ATTEMPTS never gets a chance to run. We observed worker
processes wedged for over an hour on a single recv() syscall before the
build was killed manually.

Re-introduce the CVE_SOCKET_TIMEOUT variable (removed in commit
d6d94eed1e "cve-update-nvd2-native: remove unused variable
CVE_SOCKET_TIMEOUT" as it was a leftover from the JSON 1.1 feed) and
plumb it through update_db_file() and nvd_request_next() so it is
actually honoured by urlopen(). The default of 60 seconds matches the
prior historical default; users behind slow proxies may raise it.

With the timeout in place, a stalled NVD endpoint produces a clean
exception, the retry loop runs, and after CVE_DB_UPDATE_ATTEMPTS
failures the task returns False and the build falls back to the
previously cached database (bb.warn, not a hard error).

Signed-off-by: Mehmet Fide <mehmet.fide@screeningeagle.com>
---
 meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
index 32a14a932b..271679b7bd 100644
--- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
+++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
@@ -34,6 +34,12 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000"
 # Number of attempts for each http query to nvd server before giving up
 CVE_DB_UPDATE_ATTEMPTS ?= "5"
 
+# Per-request socket timeout (seconds) for HTTP queries to the NVD server.
+# Without this, urllib uses the global default (None) and a stalled connection
+# can block the do_fetch task indefinitely, preventing the retry loop driven
+# by CVE_DB_UPDATE_ATTEMPTS from ever running.
+CVE_SOCKET_TIMEOUT ?= "60"
+
 CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}"
 CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock"
 CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"
@@ -134,7 +140,7 @@ def cleanup_db_download(db_file, db_tmp_file):
 def nvd_request_wait(attempt, min_wait):
     return min ( ( (2 * attempt) + min_wait ) , 30)
 
-def nvd_request_next(url, attempts, api_key, args, min_wait):
+def nvd_request_next(url, attempts, api_key, args, min_wait, timeout):
     """
     Request next part of the NVD database
     NVD API documentation: https://nvd.nist.gov/developers/vulnerabilities
@@ -153,7 +159,7 @@ def nvd_request_next(url, attempts, api_key, args, min_wait):
 
     for attempt in range(attempts):
         try:
-            r = urllib.request.urlopen(request)
+            r = urllib.request.urlopen(request, timeout=timeout)
 
             if (r.headers['content-encoding'] == 'gzip'):
                 buf = r.read()
@@ -216,6 +222,7 @@ def update_db_file(db_tmp_file, d, database_time):
         url = d.getVar("NVDCVE_URL")
         api_key = d.getVar("NVDCVE_API_KEY") or None
         attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS"))
+        timeout = int(d.getVar("CVE_SOCKET_TIMEOUT"))
 
         # Recommended by NVD
         wait_time = 6
@@ -224,7 +231,7 @@ def update_db_file(db_tmp_file, d, database_time):
 
         while True:
             req_args['startIndex'] = index
-            raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time)
+            raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time, timeout)
             if raw_data is None:
                 # We haven't managed to download data
                 return False
-- 
2.54.0

Re: [OE-core][PATCH][walnascar] cve-update-nvd2-native: re-introduce CVE_SOCKET_TIMEOUT to bound urlopen()

[Yoann Congal]

On Tue May 26, 2026 at 10:05 AM CEST, Mehmet Fide via lists.openembedded.org wrote:
> From: Mehmet Fide <mehmet.fide@screeningeagle.com>
>
> The fetch task calls urllib.request.urlopen() with no timeout argument, so
> when an NVD endpoint accepts the TCP connection but stops sending data,
> the call blocks forever and the existing retry loop driven by
> CVE_DB_UPDATE_ATTEMPTS never gets a chance to run. We observed worker
> processes wedged for over an hour on a single recv() syscall before the
> build was killed manually.
>
> Re-introduce the CVE_SOCKET_TIMEOUT variable (removed in commit
> d6d94eed1e "cve-update-nvd2-native: remove unused variable
> CVE_SOCKET_TIMEOUT" as it was a leftover from the JSON 1.1 feed) and
> plumb it through update_db_file() and nvd_request_next() so it is
> actually honoured by urlopen(). The default of 60 seconds matches the
> prior historical default; users behind slow proxies may raise it.
>
> With the timeout in place, a stalled NVD endpoint produces a clean
> exception, the retry loop runs, and after CVE_DB_UPDATE_ATTEMPTS
> failures the task returns False and the build falls back to the
> previously cached database (bb.warn, not a hard error).
>
> Signed-off-by: Mehmet Fide <mehmet.fide@screeningeagle.com>

Hello,

Walnascar is EOL so we can't really take this patch, sorry.

That said, given the recent NVD API glitches, it does look desirable on
the last supported stable using the NVD API: scarthgap. Can you please
rebase & send your patch there?

Thanks!

> ---
>  meta/recipes-core/meta/cve-update-nvd2-native.bb | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> index 32a14a932b..271679b7bd 100644
> --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
> +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
> @@ -34,6 +34,12 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000"
>  # Number of attempts for each http query to nvd server before giving up
>  CVE_DB_UPDATE_ATTEMPTS ?= "5"
>  
> +# Per-request socket timeout (seconds) for HTTP queries to the NVD server.
> +# Without this, urllib uses the global default (None) and a stalled connection
> +# can block the do_fetch task indefinitely, preventing the retry loop driven
> +# by CVE_DB_UPDATE_ATTEMPTS from ever running.
> +CVE_SOCKET_TIMEOUT ?= "60"
> +
>  CVE_CHECK_DB_DLDIR_FILE ?= "${DL_DIR}/CVE_CHECK2/${CVE_CHECK_DB_FILENAME}"
>  CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock"
>  CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"
> @@ -134,7 +140,7 @@ def cleanup_db_download(db_file, db_tmp_file):
>  def nvd_request_wait(attempt, min_wait):
>      return min ( ( (2 * attempt) + min_wait ) , 30)
>  
> -def nvd_request_next(url, attempts, api_key, args, min_wait):
> +def nvd_request_next(url, attempts, api_key, args, min_wait, timeout):
>      """
>      Request next part of the NVD database
>      NVD API documentation: https://nvd.nist.gov/developers/vulnerabilities
> @@ -153,7 +159,7 @@ def nvd_request_next(url, attempts, api_key, args, min_wait):
>  
>      for attempt in range(attempts):
>          try:
> -            r = urllib.request.urlopen(request)
> +            r = urllib.request.urlopen(request, timeout=timeout)
>  
>              if (r.headers['content-encoding'] == 'gzip'):
>                  buf = r.read()
> @@ -216,6 +222,7 @@ def update_db_file(db_tmp_file, d, database_time):
>          url = d.getVar("NVDCVE_URL")
>          api_key = d.getVar("NVDCVE_API_KEY") or None
>          attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS"))
> +        timeout = int(d.getVar("CVE_SOCKET_TIMEOUT"))
>  
>          # Recommended by NVD
>          wait_time = 6
> @@ -224,7 +231,7 @@ def update_db_file(db_tmp_file, d, database_time):
>  
>          while True:
>              req_args['startIndex'] = index
> -            raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time)
> +            raw_data = nvd_request_next(url, attempts, api_key, req_args, wait_time, timeout)
>              if raw_data is None:
>                  # We haven't managed to download data
>                  return False


-- 
Yoann Congal
Smile ECS