* [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener @ 2026-09-28 12:50 ecordonnier 2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier 2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier 0 siblings, 2 replies; 6+ messages in thread From: ecordonnier @ 2026-09-28 12:50 UTC (permalink / raw) To: bitbake-devel; +Cc: Etienne Cordonnier From: Etienne Cordonnier <ecordonnier@snap.com> This is the v2 of the patch originally sent as "fetch/wget: Reuse the checkstatus() urllib opener". The patch has been split in 2 individual patches, as requested in the code review. The sum of the changes in the 2 patches is bytes-identical to the original patch. Etienne Cordonnier (2): fetch/wget: Have checkstatus() capture connection_cache, not fetch fetch/wget: Reuse the checkstatus() urllib opener lib/bb/fetch/__init__.py | 2 ++ lib/bb/fetch/wget.py | 45 ++++++++++++++++++++++++---------------- 2 files changed, 29 insertions(+), 18 deletions(-) -- 2.43.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch 2026-09-28 12:50 [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier @ 2026-09-28 12:50 ` ecordonnier 2026-11-16 19:51 ` Etienne Cordonnier 2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier 1 sibling, 1 reply; 6+ messages in thread From: ecordonnier @ 2026-09-28 12:50 UTC (permalink / raw) To: bitbake-devel; +Cc: Etienne Cordonnier From: Etienne Cordonnier <ecordonnier@snap.com> checkstatus()'s nested handler classes referenced fetch.connection_cache, so their methods closed over the fetch parameter itself rather than just the connection cache. Capture connection_cache in a local variable instead and use that everywhere. No behaviour change; this makes the next commit, which caches the urllib opener built from these handlers, safe: a cached opener's closures no longer keep the Fetch object that built it alive. AI-Generated: Uses Claude Code (Claude Opus 5.5) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> --- lib/bb/fetch/wget.py | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py index 972c84048..fe2de25d3 100644 --- a/lib/bb/fetch/wget.py +++ b/lib/bb/fetch/wget.py @@ -178,23 +178,24 @@ class Wget(FetchMethod): def checkstatus(self, fetch, ud, d, try_again=True): check_certs = self.check_certs(d) newenv = bb.fetch.get_fetcher_environment(d) + connection_cache = fetch.connection_cache class HTTPConnectionCache(http.client.HTTPConnection): def cache_id(self): return None - if fetch.connection_cache: + if connection_cache: def connect(self): """Connect to the host and port specified in __init__.""" - sock = fetch.connection_cache.get_connection( + sock = connection_cache.get_connection( self.host, self.port, self.cache_id()) if sock: self.sock = sock else: self.sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address) - fetch.connection_cache.add_connection( + connection_cache.add_connection( self.host, self.port, self.sock, self.cache_id()) if self._tunnel_host: @@ -206,17 +207,17 @@ class Wget(FetchMethod): newenv.get("SSL_CERT_FILE"), self._tunnel_host, self._tunnel_port) - if fetch.connection_cache: + if connection_cache: def connect(self): """Reuse an established TLS connection when available.""" - sock = fetch.connection_cache.get_connection( + sock = connection_cache.get_connection( self.host, self.port, self.cache_id()) if sock: self.sock = sock else: super().connect() - fetch.connection_cache.add_connection( + connection_cache.add_connection( self.host, self.port, self.sock, self.cache_id()) class CacheHTTPHandler(urllib.request.HTTPHandler): @@ -252,7 +253,7 @@ class Wget(FetchMethod): # request. # Don't close connection when connection_cache is enabled, - if fetch.connection_cache is None: + if connection_cache is None: headers["Connection"] = "close" else: headers["Connection"] = "Keep-Alive" # Works for HTTP/1.0 @@ -282,8 +283,8 @@ class Wget(FetchMethod): # with the dead connection removed from the cache. # If it still fails, we give up, which can happen for bad # HTTP proxy settings. - if fetch.connection_cache: - fetch.connection_cache.remove_connection( + if connection_cache: + connection_cache.remove_connection( h.host, h.port, h.cache_id()) h.close() raise @@ -316,9 +317,9 @@ class Wget(FetchMethod): resp.msg = r.reason # Close connection when server request it. - if fetch.connection_cache is not None: + if connection_cache is not None: if 'Connection' in r.msg and r.msg['Connection'] == 'close': - fetch.connection_cache.remove_connection( + connection_cache.remove_connection( h.host, h.port, h.cache_id()) return resp -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch 2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier @ 2026-11-16 19:51 ` Etienne Cordonnier 2026-11-17 8:53 ` Paul Barker 0 siblings, 1 reply; 6+ messages in thread From: Etienne Cordonnier @ 2026-11-16 19:51 UTC (permalink / raw) To: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 5128 bytes --] Hi, is the patch OK? I see that it has been in master-next for a while. Is there some potential issue I should address? Étienne On Mon, Sep 28, 2026 at 2:51 PM <ecordonnier@snap.com> wrote: > From: Etienne Cordonnier <ecordonnier@snap.com> > > checkstatus()'s nested handler classes referenced fetch.connection_cache, > so their methods closed over the fetch parameter itself rather than just > the connection cache. Capture connection_cache in a local variable instead > and use that everywhere. No behaviour change; this makes the next commit, > which caches the urllib opener built from these handlers, safe: a cached > opener's closures no longer keep the Fetch object that built it alive. > > AI-Generated: Uses Claude Code (Claude Opus 5.5) > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> > --- > lib/bb/fetch/wget.py | 23 ++++++++++++----------- > 1 file changed, 12 insertions(+), 11 deletions(-) > > diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py > index 972c84048..fe2de25d3 100644 > --- a/lib/bb/fetch/wget.py > +++ b/lib/bb/fetch/wget.py > @@ -178,23 +178,24 @@ class Wget(FetchMethod): > def checkstatus(self, fetch, ud, d, try_again=True): > check_certs = self.check_certs(d) > newenv = bb.fetch.get_fetcher_environment(d) > + connection_cache = fetch.connection_cache > > class HTTPConnectionCache(http.client.HTTPConnection): > def cache_id(self): > return None > > - if fetch.connection_cache: > + if connection_cache: > def connect(self): > """Connect to the host and port specified in > __init__.""" > > - sock = fetch.connection_cache.get_connection( > + sock = connection_cache.get_connection( > self.host, self.port, self.cache_id()) > if sock: > self.sock = sock > else: > self.sock = socket.create_connection((self.host, > self.port), > self.timeout, self.source_address) > - fetch.connection_cache.add_connection( > + connection_cache.add_connection( > self.host, self.port, self.sock, > self.cache_id()) > > if self._tunnel_host: > @@ -206,17 +207,17 @@ class Wget(FetchMethod): > newenv.get("SSL_CERT_FILE"), > self._tunnel_host, self._tunnel_port) > > - if fetch.connection_cache: > + if connection_cache: > def connect(self): > """Reuse an established TLS connection when > available.""" > > - sock = fetch.connection_cache.get_connection( > + sock = connection_cache.get_connection( > self.host, self.port, self.cache_id()) > if sock: > self.sock = sock > else: > super().connect() > - fetch.connection_cache.add_connection( > + connection_cache.add_connection( > self.host, self.port, self.sock, > self.cache_id()) > > class CacheHTTPHandler(urllib.request.HTTPHandler): > @@ -252,7 +253,7 @@ class Wget(FetchMethod): > # request. > > # Don't close connection when connection_cache is enabled, > - if fetch.connection_cache is None: > + if connection_cache is None: > headers["Connection"] = "close" > else: > headers["Connection"] = "Keep-Alive" # Works for > HTTP/1.0 > @@ -282,8 +283,8 @@ class Wget(FetchMethod): > # with the dead connection removed from the cache. > # If it still fails, we give up, which can happen for > bad > # HTTP proxy settings. > - if fetch.connection_cache: > - fetch.connection_cache.remove_connection( > + if connection_cache: > + connection_cache.remove_connection( > h.host, h.port, h.cache_id()) > h.close() > raise > @@ -316,9 +317,9 @@ class Wget(FetchMethod): > resp.msg = r.reason > > # Close connection when server request it. > - if fetch.connection_cache is not None: > + if connection_cache is not None: > if 'Connection' in r.msg and r.msg['Connection'] == > 'close': > - fetch.connection_cache.remove_connection( > + connection_cache.remove_connection( > h.host, h.port, h.cache_id()) > > return resp > -- > 2.43.0 > > [-- Attachment #2: Type: text/html, Size: 6576 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch 2026-11-16 19:51 ` Etienne Cordonnier @ 2026-11-17 8:53 ` Paul Barker 0 siblings, 0 replies; 6+ messages in thread From: Paul Barker @ 2026-11-17 8:53 UTC (permalink / raw) To: Etienne Cordonnier, bitbake-devel On Mon Nov 16, 2026 at 7:51 PM GMT, Etienne Cordonnier wrote: > Hi, > is the patch OK? I see that it has been in master-next for a while. Is > there some potential issue I should address? Hi Étienne, Unfortunately we have limited bandwidth for reviewing changes and there is a lot of work in progress. Apologies that this got stuck in the queue. This patch looks fine, I will take a look at the second patch in the series and give some feedback. > > On Mon, Sep 28, 2026 at 2:51 PM <ecordonnier@snap.com> wrote: > >> From: Etienne Cordonnier <ecordonnier@snap.com> >> >> checkstatus()'s nested handler classes referenced fetch.connection_cache, >> so their methods closed over the fetch parameter itself rather than just >> the connection cache. Capture connection_cache in a local variable instead >> and use that everywhere. No behaviour change; this makes the next commit, >> which caches the urllib opener built from these handlers, safe: a cached >> opener's closures no longer keep the Fetch object that built it alive. >> >> AI-Generated: Uses Claude Code (Claude Opus 5.5) >> Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> >> --- >> lib/bb/fetch/wget.py | 23 ++++++++++++----------- >> 1 file changed, 12 insertions(+), 11 deletions(-) >> >> diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py >> index 972c84048..fe2de25d3 100644 >> --- a/lib/bb/fetch/wget.py >> +++ b/lib/bb/fetch/wget.py >> @@ -178,23 +178,24 @@ class Wget(FetchMethod): >> def checkstatus(self, fetch, ud, d, try_again=True): >> check_certs = self.check_certs(d) >> newenv = bb.fetch.get_fetcher_environment(d) >> + connection_cache = fetch.connection_cache >> >> class HTTPConnectionCache(http.client.HTTPConnection): >> def cache_id(self): >> return None >> >> - if fetch.connection_cache: >> + if connection_cache: >> def connect(self): >> """Connect to the host and port specified in >> __init__.""" >> >> - sock = fetch.connection_cache.get_connection( >> + sock = connection_cache.get_connection( >> self.host, self.port, self.cache_id()) >> if sock: >> self.sock = sock >> else: >> self.sock = socket.create_connection((self.host, >> self.port), >> self.timeout, self.source_address) >> - fetch.connection_cache.add_connection( >> + connection_cache.add_connection( >> self.host, self.port, self.sock, >> self.cache_id()) >> >> if self._tunnel_host: >> @@ -206,17 +207,17 @@ class Wget(FetchMethod): >> newenv.get("SSL_CERT_FILE"), >> self._tunnel_host, self._tunnel_port) >> >> - if fetch.connection_cache: >> + if connection_cache: >> def connect(self): >> """Reuse an established TLS connection when >> available.""" >> >> - sock = fetch.connection_cache.get_connection( >> + sock = connection_cache.get_connection( >> self.host, self.port, self.cache_id()) >> if sock: >> self.sock = sock >> else: >> super().connect() >> - fetch.connection_cache.add_connection( >> + connection_cache.add_connection( >> self.host, self.port, self.sock, >> self.cache_id()) >> >> class CacheHTTPHandler(urllib.request.HTTPHandler): >> @@ -252,7 +253,7 @@ class Wget(FetchMethod): >> # request. >> >> # Don't close connection when connection_cache is enabled, >> - if fetch.connection_cache is None: >> + if connection_cache is None: >> headers["Connection"] = "close" >> else: >> headers["Connection"] = "Keep-Alive" # Works for >> HTTP/1.0 >> @@ -282,8 +283,8 @@ class Wget(FetchMethod): >> # with the dead connection removed from the cache. >> # If it still fails, we give up, which can happen for >> bad >> # HTTP proxy settings. >> - if fetch.connection_cache: >> - fetch.connection_cache.remove_connection( >> + if connection_cache: >> + connection_cache.remove_connection( >> h.host, h.port, h.cache_id()) >> h.close() >> raise >> @@ -316,9 +317,9 @@ class Wget(FetchMethod): >> resp.msg = r.reason >> >> # Close connection when server request it. >> - if fetch.connection_cache is not None: >> + if connection_cache is not None: >> if 'Connection' in r.msg and r.msg['Connection'] == >> 'close': >> - fetch.connection_cache.remove_connection( >> + connection_cache.remove_connection( >> h.host, h.port, h.cache_id()) >> >> return resp >> -- >> 2.43.0 >> >> Best regards, -- Paul Barker ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener 2026-09-28 12:50 [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier 2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier @ 2026-09-28 12:50 ` ecordonnier 2026-11-17 8:58 ` Paul Barker 1 sibling, 1 reply; 6+ messages in thread From: ecordonnier @ 2026-09-28 12:50 UTC (permalink / raw) To: bitbake-devel; +Cc: Etienne Cordonnier From: Etienne Cordonnier <ecordonnier@snap.com> checkstatus() builds a new urllib opener with its redirect, proxy and connection cache handlers on every call, although sstate.bbclass calls it for every object in an sstate mirror check with the same settings. Keep the opener on the FetchConnectionCache, keyed on the certificate checking setting and the fetcher environment, and rebuild it only when those change. A connection cache is only used by one thread at a time, so no locking is needed. Checking 2,114 objects on the Yocto sstate mirror with BB_NUMBER_THREADS = "8" reduced the time to task execution from 26.9s to 25.1s on average. AI-Generated: Uses Claude Code (Claude Opus 5.5) Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> --- lib/bb/fetch/__init__.py | 2 ++ lib/bb/fetch/wget.py | 22 +++++++++++++++------- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py index 55ab710f3..0eb660685 100644 --- a/lib/bb/fetch/__init__.py +++ b/lib/bb/fetch/__init__.py @@ -2090,6 +2090,8 @@ class FetchConnectionCache(object): """ def __init__(self): self.cache = {} + # urllib openers built by the wget fetcher, see Wget.checkstatus() + self.opener_cache = {} def get_connection_name(self, host, port, connection_id=None): if connection_id is None: diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py index fe2de25d3..9cb4a4f32 100644 --- a/lib/bb/fetch/wget.py +++ b/lib/bb/fetch/wget.py @@ -435,13 +435,21 @@ class Wget(FetchMethod): # to scope the changes to the build_opener request, which is when the # environment lookups happen. with bb.utils.environment(**newenv): - context = self.ssl_context(d) - handlers = [FixedHTTPRedirectHandler, - HTTPMethodFallback, - urllib.request.ProxyHandler(), - CacheHTTPHandler(), - CacheHTTPSHandler(context=context)] - opener = urllib.request.build_opener(*handlers) + opener = None + if connection_cache is not None: + opener_key = (check_certs, tuple(sorted(newenv.items()))) + opener = connection_cache.opener_cache.get(opener_key) + + if opener is None: + context = self.ssl_context(d) + handlers = [FixedHTTPRedirectHandler, + HTTPMethodFallback, + urllib.request.ProxyHandler(), + CacheHTTPHandler(), + CacheHTTPSHandler(context=context)] + opener = urllib.request.build_opener(*handlers) + if connection_cache is not None: + connection_cache.opener_cache[opener_key] = opener try: parts = urllib.parse.urlparse(ud.url.split(";")[0]) -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener 2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier @ 2026-11-17 8:58 ` Paul Barker 0 siblings, 0 replies; 6+ messages in thread From: Paul Barker @ 2026-11-17 8:58 UTC (permalink / raw) To: ecordonnier, bitbake-devel On Mon Sep 28, 2026 at 1:50 PM BST, ecordonnier wrote: > From: Etienne Cordonnier <ecordonnier@snap.com> > > checkstatus() builds a new urllib opener with its redirect, proxy and > connection cache handlers on every call, although sstate.bbclass calls it > for every object in an sstate mirror check with the same settings. Keep > the opener on the FetchConnectionCache, keyed on the certificate checking > setting and the fetcher environment, and rebuild it only when those > change. A connection cache is only used by one thread at a time, so no > locking is needed. > > Checking 2,114 objects on the Yocto sstate mirror with BB_NUMBER_THREADS = "8" > reduced the time to task execution from 26.9s to 25.1s on average. This is about a 7% performance improvement, so would be good to have. The question is, what state do the urlllib opener and the handler instances maintain across requests? Is there risk of credentials or headers intended for one host being inappropriately sent with the request to a different host? Someone needs to look at this and confirm that it doesn't introduce any security risks. > > AI-Generated: Uses Claude Code (Claude Opus 5.5) > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com> > --- > lib/bb/fetch/__init__.py | 2 ++ > lib/bb/fetch/wget.py | 22 +++++++++++++++------- > 2 files changed, 17 insertions(+), 7 deletions(-) > > diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py > index 55ab710f3..0eb660685 100644 > --- a/lib/bb/fetch/__init__.py > +++ b/lib/bb/fetch/__init__.py > @@ -2090,6 +2090,8 @@ class FetchConnectionCache(object): > """ > def __init__(self): > self.cache = {} > + # urllib openers built by the wget fetcher, see Wget.checkstatus() > + self.opener_cache = {} > > def get_connection_name(self, host, port, connection_id=None): > if connection_id is None: > diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py > index fe2de25d3..9cb4a4f32 100644 > --- a/lib/bb/fetch/wget.py > +++ b/lib/bb/fetch/wget.py > @@ -435,13 +435,21 @@ class Wget(FetchMethod): > # to scope the changes to the build_opener request, which is when the > # environment lookups happen. > with bb.utils.environment(**newenv): > - context = self.ssl_context(d) > - handlers = [FixedHTTPRedirectHandler, > - HTTPMethodFallback, > - urllib.request.ProxyHandler(), > - CacheHTTPHandler(), > - CacheHTTPSHandler(context=context)] > - opener = urllib.request.build_opener(*handlers) > + opener = None > + if connection_cache is not None: > + opener_key = (check_certs, tuple(sorted(newenv.items()))) > + opener = connection_cache.opener_cache.get(opener_key) > + > + if opener is None: > + context = self.ssl_context(d) > + handlers = [FixedHTTPRedirectHandler, > + HTTPMethodFallback, > + urllib.request.ProxyHandler(), > + CacheHTTPHandler(), > + CacheHTTPSHandler(context=context)] > + opener = urllib.request.build_opener(*handlers) > + if connection_cache is not None: > + connection_cache.opener_cache[opener_key] = opener > > try: > parts = urllib.parse.urlparse(ud.url.split(";")[0]) Best regards, -- Paul Barker ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-17 8:58 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-09-28 12:50 [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier 2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier 2026-11-16 19:51 ` Etienne Cordonnier 2026-11-17 8:53 ` Paul Barker 2026-09-28 12:50 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier 2026-11-17 8:58 ` Paul Barker
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.