* [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener
@ 2026-09-28 12:50 ecordonnier
2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier
2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
0 siblings, 2 replies; 6+ messages in thread
From: ecordonnier @ 2026-09-28 12:50 UTC (permalink / raw)
To: bitbake-devel; +Cc: Etienne Cordonnier
From: Etienne Cordonnier <ecordonnier@snap.com>
This is the v2 of the patch originally sent as "fetch/wget: Reuse the checkstatus() urllib opener".
The patch has been split in 2 individual patches, as requested in the code review.
The sum of the changes in the 2 patches is bytes-identical to the original patch.
Etienne Cordonnier (2):
fetch/wget: Have checkstatus() capture connection_cache, not fetch
fetch/wget: Reuse the checkstatus() urllib opener
lib/bb/fetch/__init__.py | 2 ++
lib/bb/fetch/wget.py | 45 ++++++++++++++++++++++++----------------
2 files changed, 29 insertions(+), 18 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch
2026-09-28 12:50 [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
@ 2026-09-28 12:50 ` ecordonnier
2026-11-16 19:51 ` Etienne Cordonnier
2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
1 sibling, 1 reply; 6+ messages in thread
From: ecordonnier @ 2026-09-28 12:50 UTC (permalink / raw)
To: bitbake-devel; +Cc: Etienne Cordonnier
From: Etienne Cordonnier <ecordonnier@snap.com>
checkstatus()'s nested handler classes referenced fetch.connection_cache,
so their methods closed over the fetch parameter itself rather than just
the connection cache. Capture connection_cache in a local variable instead
and use that everywhere. No behaviour change; this makes the next commit,
which caches the urllib opener built from these handlers, safe: a cached
opener's closures no longer keep the Fetch object that built it alive.
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
lib/bb/fetch/wget.py | 23 ++++++++++++-----------
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py
index 972c84048..fe2de25d3 100644
--- a/lib/bb/fetch/wget.py
+++ b/lib/bb/fetch/wget.py
@@ -178,23 +178,24 @@ class Wget(FetchMethod):
def checkstatus(self, fetch, ud, d, try_again=True):
check_certs = self.check_certs(d)
newenv = bb.fetch.get_fetcher_environment(d)
+ connection_cache = fetch.connection_cache
class HTTPConnectionCache(http.client.HTTPConnection):
def cache_id(self):
return None
- if fetch.connection_cache:
+ if connection_cache:
def connect(self):
"""Connect to the host and port specified in __init__."""
- sock = fetch.connection_cache.get_connection(
+ sock = connection_cache.get_connection(
self.host, self.port, self.cache_id())
if sock:
self.sock = sock
else:
self.sock = socket.create_connection((self.host, self.port),
self.timeout, self.source_address)
- fetch.connection_cache.add_connection(
+ connection_cache.add_connection(
self.host, self.port, self.sock, self.cache_id())
if self._tunnel_host:
@@ -206,17 +207,17 @@ class Wget(FetchMethod):
newenv.get("SSL_CERT_FILE"),
self._tunnel_host, self._tunnel_port)
- if fetch.connection_cache:
+ if connection_cache:
def connect(self):
"""Reuse an established TLS connection when available."""
- sock = fetch.connection_cache.get_connection(
+ sock = connection_cache.get_connection(
self.host, self.port, self.cache_id())
if sock:
self.sock = sock
else:
super().connect()
- fetch.connection_cache.add_connection(
+ connection_cache.add_connection(
self.host, self.port, self.sock, self.cache_id())
class CacheHTTPHandler(urllib.request.HTTPHandler):
@@ -252,7 +253,7 @@ class Wget(FetchMethod):
# request.
# Don't close connection when connection_cache is enabled,
- if fetch.connection_cache is None:
+ if connection_cache is None:
headers["Connection"] = "close"
else:
headers["Connection"] = "Keep-Alive" # Works for HTTP/1.0
@@ -282,8 +283,8 @@ class Wget(FetchMethod):
# with the dead connection removed from the cache.
# If it still fails, we give up, which can happen for bad
# HTTP proxy settings.
- if fetch.connection_cache:
- fetch.connection_cache.remove_connection(
+ if connection_cache:
+ connection_cache.remove_connection(
h.host, h.port, h.cache_id())
h.close()
raise
@@ -316,9 +317,9 @@ class Wget(FetchMethod):
resp.msg = r.reason
# Close connection when server request it.
- if fetch.connection_cache is not None:
+ if connection_cache is not None:
if 'Connection' in r.msg and r.msg['Connection'] == 'close':
- fetch.connection_cache.remove_connection(
+ connection_cache.remove_connection(
h.host, h.port, h.cache_id())
return resp
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener
2026-09-28 12:50 [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier
@ 2026-09-28 12:50 ` ecordonnier
2026-11-17 8:58 ` Paul Barker
1 sibling, 1 reply; 6+ messages in thread
From: ecordonnier @ 2026-09-28 12:50 UTC (permalink / raw)
To: bitbake-devel; +Cc: Etienne Cordonnier
From: Etienne Cordonnier <ecordonnier@snap.com>
checkstatus() builds a new urllib opener with its redirect, proxy and
connection cache handlers on every call, although sstate.bbclass calls it
for every object in an sstate mirror check with the same settings. Keep
the opener on the FetchConnectionCache, keyed on the certificate checking
setting and the fetcher environment, and rebuild it only when those
change. A connection cache is only used by one thread at a time, so no
locking is needed.
Checking 2,114 objects on the Yocto sstate mirror with BB_NUMBER_THREADS = "8"
reduced the time to task execution from 26.9s to 25.1s on average.
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
lib/bb/fetch/__init__.py | 2 ++
lib/bb/fetch/wget.py | 22 +++++++++++++++-------
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 55ab710f3..0eb660685 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -2090,6 +2090,8 @@ class FetchConnectionCache(object):
"""
def __init__(self):
self.cache = {}
+ # urllib openers built by the wget fetcher, see Wget.checkstatus()
+ self.opener_cache = {}
def get_connection_name(self, host, port, connection_id=None):
if connection_id is None:
diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py
index fe2de25d3..9cb4a4f32 100644
--- a/lib/bb/fetch/wget.py
+++ b/lib/bb/fetch/wget.py
@@ -435,13 +435,21 @@ class Wget(FetchMethod):
# to scope the changes to the build_opener request, which is when the
# environment lookups happen.
with bb.utils.environment(**newenv):
- context = self.ssl_context(d)
- handlers = [FixedHTTPRedirectHandler,
- HTTPMethodFallback,
- urllib.request.ProxyHandler(),
- CacheHTTPHandler(),
- CacheHTTPSHandler(context=context)]
- opener = urllib.request.build_opener(*handlers)
+ opener = None
+ if connection_cache is not None:
+ opener_key = (check_certs, tuple(sorted(newenv.items())))
+ opener = connection_cache.opener_cache.get(opener_key)
+
+ if opener is None:
+ context = self.ssl_context(d)
+ handlers = [FixedHTTPRedirectHandler,
+ HTTPMethodFallback,
+ urllib.request.ProxyHandler(),
+ CacheHTTPHandler(),
+ CacheHTTPSHandler(context=context)]
+ opener = urllib.request.build_opener(*handlers)
+ if connection_cache is not None:
+ connection_cache.opener_cache[opener_key] = opener
try:
parts = urllib.parse.urlparse(ud.url.split(";")[0])
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch
2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier
@ 2026-11-16 19:51 ` Etienne Cordonnier
2026-11-17 8:53 ` Paul Barker
0 siblings, 1 reply; 6+ messages in thread
From: Etienne Cordonnier @ 2026-11-16 19:51 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 5128 bytes --]
Hi,
is the patch OK? I see that it has been in master-next for a while. Is
there some potential issue I should address?
Étienne
On Mon, Sep 28, 2026 at 2:51 PM <ecordonnier@snap.com> wrote:
> From: Etienne Cordonnier <ecordonnier@snap.com>
>
> checkstatus()'s nested handler classes referenced fetch.connection_cache,
> so their methods closed over the fetch parameter itself rather than just
> the connection cache. Capture connection_cache in a local variable instead
> and use that everywhere. No behaviour change; this makes the next commit,
> which caches the urllib opener built from these handlers, safe: a cached
> opener's closures no longer keep the Fetch object that built it alive.
>
> AI-Generated: Uses Claude Code (Claude Opus 5.5)
> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> ---
> lib/bb/fetch/wget.py | 23 ++++++++++++-----------
> 1 file changed, 12 insertions(+), 11 deletions(-)
>
> diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py
> index 972c84048..fe2de25d3 100644
> --- a/lib/bb/fetch/wget.py
> +++ b/lib/bb/fetch/wget.py
> @@ -178,23 +178,24 @@ class Wget(FetchMethod):
> def checkstatus(self, fetch, ud, d, try_again=True):
> check_certs = self.check_certs(d)
> newenv = bb.fetch.get_fetcher_environment(d)
> + connection_cache = fetch.connection_cache
>
> class HTTPConnectionCache(http.client.HTTPConnection):
> def cache_id(self):
> return None
>
> - if fetch.connection_cache:
> + if connection_cache:
> def connect(self):
> """Connect to the host and port specified in
> __init__."""
>
> - sock = fetch.connection_cache.get_connection(
> + sock = connection_cache.get_connection(
> self.host, self.port, self.cache_id())
> if sock:
> self.sock = sock
> else:
> self.sock = socket.create_connection((self.host,
> self.port),
> self.timeout, self.source_address)
> - fetch.connection_cache.add_connection(
> + connection_cache.add_connection(
> self.host, self.port, self.sock,
> self.cache_id())
>
> if self._tunnel_host:
> @@ -206,17 +207,17 @@ class Wget(FetchMethod):
> newenv.get("SSL_CERT_FILE"),
> self._tunnel_host, self._tunnel_port)
>
> - if fetch.connection_cache:
> + if connection_cache:
> def connect(self):
> """Reuse an established TLS connection when
> available."""
>
> - sock = fetch.connection_cache.get_connection(
> + sock = connection_cache.get_connection(
> self.host, self.port, self.cache_id())
> if sock:
> self.sock = sock
> else:
> super().connect()
> - fetch.connection_cache.add_connection(
> + connection_cache.add_connection(
> self.host, self.port, self.sock,
> self.cache_id())
>
> class CacheHTTPHandler(urllib.request.HTTPHandler):
> @@ -252,7 +253,7 @@ class Wget(FetchMethod):
> # request.
>
> # Don't close connection when connection_cache is enabled,
> - if fetch.connection_cache is None:
> + if connection_cache is None:
> headers["Connection"] = "close"
> else:
> headers["Connection"] = "Keep-Alive" # Works for
> HTTP/1.0
> @@ -282,8 +283,8 @@ class Wget(FetchMethod):
> # with the dead connection removed from the cache.
> # If it still fails, we give up, which can happen for
> bad
> # HTTP proxy settings.
> - if fetch.connection_cache:
> - fetch.connection_cache.remove_connection(
> + if connection_cache:
> + connection_cache.remove_connection(
> h.host, h.port, h.cache_id())
> h.close()
> raise
> @@ -316,9 +317,9 @@ class Wget(FetchMethod):
> resp.msg = r.reason
>
> # Close connection when server request it.
> - if fetch.connection_cache is not None:
> + if connection_cache is not None:
> if 'Connection' in r.msg and r.msg['Connection'] ==
> 'close':
> - fetch.connection_cache.remove_connection(
> + connection_cache.remove_connection(
> h.host, h.port, h.cache_id())
>
> return resp
> --
> 2.43.0
>
>
[-- Attachment #2: Type: text/html, Size: 6576 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch
2026-11-16 19:51 ` Etienne Cordonnier
@ 2026-11-17 8:53 ` Paul Barker
0 siblings, 0 replies; 6+ messages in thread
From: Paul Barker @ 2026-11-17 8:53 UTC (permalink / raw)
To: Etienne Cordonnier, bitbake-devel
On Mon Nov 16, 2026 at 7:51 PM GMT, Etienne Cordonnier wrote:
> Hi,
> is the patch OK? I see that it has been in master-next for a while. Is
> there some potential issue I should address?
Hi Étienne,
Unfortunately we have limited bandwidth for reviewing changes and there
is a lot of work in progress. Apologies that this got stuck in the
queue.
This patch looks fine, I will take a look at the second patch in the
series and give some feedback.
>
> On Mon, Sep 28, 2026 at 2:51 PM <ecordonnier@snap.com> wrote:
>
>> From: Etienne Cordonnier <ecordonnier@snap.com>
>>
>> checkstatus()'s nested handler classes referenced fetch.connection_cache,
>> so their methods closed over the fetch parameter itself rather than just
>> the connection cache. Capture connection_cache in a local variable instead
>> and use that everywhere. No behaviour change; this makes the next commit,
>> which caches the urllib opener built from these handlers, safe: a cached
>> opener's closures no longer keep the Fetch object that built it alive.
>>
>> AI-Generated: Uses Claude Code (Claude Opus 5.5)
>> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
>> ---
>> lib/bb/fetch/wget.py | 23 ++++++++++++-----------
>> 1 file changed, 12 insertions(+), 11 deletions(-)
>>
>> diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py
>> index 972c84048..fe2de25d3 100644
>> --- a/lib/bb/fetch/wget.py
>> +++ b/lib/bb/fetch/wget.py
>> @@ -178,23 +178,24 @@ class Wget(FetchMethod):
>> def checkstatus(self, fetch, ud, d, try_again=True):
>> check_certs = self.check_certs(d)
>> newenv = bb.fetch.get_fetcher_environment(d)
>> + connection_cache = fetch.connection_cache
>>
>> class HTTPConnectionCache(http.client.HTTPConnection):
>> def cache_id(self):
>> return None
>>
>> - if fetch.connection_cache:
>> + if connection_cache:
>> def connect(self):
>> """Connect to the host and port specified in
>> __init__."""
>>
>> - sock = fetch.connection_cache.get_connection(
>> + sock = connection_cache.get_connection(
>> self.host, self.port, self.cache_id())
>> if sock:
>> self.sock = sock
>> else:
>> self.sock = socket.create_connection((self.host,
>> self.port),
>> self.timeout, self.source_address)
>> - fetch.connection_cache.add_connection(
>> + connection_cache.add_connection(
>> self.host, self.port, self.sock,
>> self.cache_id())
>>
>> if self._tunnel_host:
>> @@ -206,17 +207,17 @@ class Wget(FetchMethod):
>> newenv.get("SSL_CERT_FILE"),
>> self._tunnel_host, self._tunnel_port)
>>
>> - if fetch.connection_cache:
>> + if connection_cache:
>> def connect(self):
>> """Reuse an established TLS connection when
>> available."""
>>
>> - sock = fetch.connection_cache.get_connection(
>> + sock = connection_cache.get_connection(
>> self.host, self.port, self.cache_id())
>> if sock:
>> self.sock = sock
>> else:
>> super().connect()
>> - fetch.connection_cache.add_connection(
>> + connection_cache.add_connection(
>> self.host, self.port, self.sock,
>> self.cache_id())
>>
>> class CacheHTTPHandler(urllib.request.HTTPHandler):
>> @@ -252,7 +253,7 @@ class Wget(FetchMethod):
>> # request.
>>
>> # Don't close connection when connection_cache is enabled,
>> - if fetch.connection_cache is None:
>> + if connection_cache is None:
>> headers["Connection"] = "close"
>> else:
>> headers["Connection"] = "Keep-Alive" # Works for
>> HTTP/1.0
>> @@ -282,8 +283,8 @@ class Wget(FetchMethod):
>> # with the dead connection removed from the cache.
>> # If it still fails, we give up, which can happen for
>> bad
>> # HTTP proxy settings.
>> - if fetch.connection_cache:
>> - fetch.connection_cache.remove_connection(
>> + if connection_cache:
>> + connection_cache.remove_connection(
>> h.host, h.port, h.cache_id())
>> h.close()
>> raise
>> @@ -316,9 +317,9 @@ class Wget(FetchMethod):
>> resp.msg = r.reason
>>
>> # Close connection when server request it.
>> - if fetch.connection_cache is not None:
>> + if connection_cache is not None:
>> if 'Connection' in r.msg and r.msg['Connection'] ==
>> 'close':
>> - fetch.connection_cache.remove_connection(
>> + connection_cache.remove_connection(
>> h.host, h.port, h.cache_id())
>>
>> return resp
>> --
>> 2.43.0
>>
>>
Best regards,
--
Paul Barker
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener
2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
@ 2026-11-17 8:58 ` Paul Barker
0 siblings, 0 replies; 6+ messages in thread
From: Paul Barker @ 2026-11-17 8:58 UTC (permalink / raw)
To: ecordonnier, bitbake-devel
On Mon Sep 28, 2026 at 1:50 PM BST, ecordonnier wrote:
> From: Etienne Cordonnier <ecordonnier@snap.com>
>
> checkstatus() builds a new urllib opener with its redirect, proxy and
> connection cache handlers on every call, although sstate.bbclass calls it
> for every object in an sstate mirror check with the same settings. Keep
> the opener on the FetchConnectionCache, keyed on the certificate checking
> setting and the fetcher environment, and rebuild it only when those
> change. A connection cache is only used by one thread at a time, so no
> locking is needed.
>
> Checking 2,114 objects on the Yocto sstate mirror with BB_NUMBER_THREADS = "8"
> reduced the time to task execution from 26.9s to 25.1s on average.
This is about a 7% performance improvement, so would be good to have.
The question is, what state do the urlllib opener and the handler
instances maintain across requests? Is there risk of credentials or
headers intended for one host being inappropriately sent with the
request to a different host?
Someone needs to look at this and confirm that it doesn't introduce any
security risks.
>
> AI-Generated: Uses Claude Code (Claude Opus 5.5)
> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> ---
> lib/bb/fetch/__init__.py | 2 ++
> lib/bb/fetch/wget.py | 22 +++++++++++++++-------
> 2 files changed, 17 insertions(+), 7 deletions(-)
>
> diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
> index 55ab710f3..0eb660685 100644
> --- a/lib/bb/fetch/__init__.py
> +++ b/lib/bb/fetch/__init__.py
> @@ -2090,6 +2090,8 @@ class FetchConnectionCache(object):
> """
> def __init__(self):
> self.cache = {}
> + # urllib openers built by the wget fetcher, see Wget.checkstatus()
> + self.opener_cache = {}
>
> def get_connection_name(self, host, port, connection_id=None):
> if connection_id is None:
> diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py
> index fe2de25d3..9cb4a4f32 100644
> --- a/lib/bb/fetch/wget.py
> +++ b/lib/bb/fetch/wget.py
> @@ -435,13 +435,21 @@ class Wget(FetchMethod):
> # to scope the changes to the build_opener request, which is when the
> # environment lookups happen.
> with bb.utils.environment(**newenv):
> - context = self.ssl_context(d)
> - handlers = [FixedHTTPRedirectHandler,
> - HTTPMethodFallback,
> - urllib.request.ProxyHandler(),
> - CacheHTTPHandler(),
> - CacheHTTPSHandler(context=context)]
> - opener = urllib.request.build_opener(*handlers)
> + opener = None
> + if connection_cache is not None:
> + opener_key = (check_certs, tuple(sorted(newenv.items())))
> + opener = connection_cache.opener_cache.get(opener_key)
> +
> + if opener is None:
> + context = self.ssl_context(d)
> + handlers = [FixedHTTPRedirectHandler,
> + HTTPMethodFallback,
> + urllib.request.ProxyHandler(),
> + CacheHTTPHandler(),
> + CacheHTTPSHandler(context=context)]
> + opener = urllib.request.build_opener(*handlers)
> + if connection_cache is not None:
> + connection_cache.opener_cache[opener_key] = opener
>
> try:
> parts = urllib.parse.urlparse(ud.url.split(";")[0])
Best regards,
--
Paul Barker
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-17 8:58 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-28 12:50 [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier
2026-11-16 19:51 ` Etienne Cordonnier
2026-11-17 8:53 ` Paul Barker
2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
2026-11-17 8:58 ` Paul Barker
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.