From: ecordonnier@snap.com
To: bitbake-devel@lists.openembedded.org
Cc: Etienne Cordonnier <ecordonnier@snap.com>
Subject: [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener
Date: Mon, 28 Sep 2026 14:50:51 +0200 [thread overview]
Message-ID: <20260928125051.2483039-3-ecordonnier@snap.com> (raw)
In-Reply-To: <20260928125051.2483039-1-ecordonnier@snap.com>
From: Etienne Cordonnier <ecordonnier@snap.com>
checkstatus() builds a new urllib opener with its redirect, proxy and
connection cache handlers on every call, although sstate.bbclass calls it
for every object in an sstate mirror check with the same settings. Keep
the opener on the FetchConnectionCache, keyed on the certificate checking
setting and the fetcher environment, and rebuild it only when those
change. A connection cache is only used by one thread at a time, so no
locking is needed.
Checking 2,114 objects on the Yocto sstate mirror with BB_NUMBER_THREADS = "8"
reduced the time to task execution from 26.9s to 25.1s on average.
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
lib/bb/fetch/__init__.py | 2 ++
lib/bb/fetch/wget.py | 22 +++++++++++++++-------
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index 55ab710f3..0eb660685 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -2090,6 +2090,8 @@ class FetchConnectionCache(object):
"""
def __init__(self):
self.cache = {}
+ # urllib openers built by the wget fetcher, see Wget.checkstatus()
+ self.opener_cache = {}
def get_connection_name(self, host, port, connection_id=None):
if connection_id is None:
diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py
index fe2de25d3..9cb4a4f32 100644
--- a/lib/bb/fetch/wget.py
+++ b/lib/bb/fetch/wget.py
@@ -435,13 +435,21 @@ class Wget(FetchMethod):
# to scope the changes to the build_opener request, which is when the
# environment lookups happen.
with bb.utils.environment(**newenv):
- context = self.ssl_context(d)
- handlers = [FixedHTTPRedirectHandler,
- HTTPMethodFallback,
- urllib.request.ProxyHandler(),
- CacheHTTPHandler(),
- CacheHTTPSHandler(context=context)]
- opener = urllib.request.build_opener(*handlers)
+ opener = None
+ if connection_cache is not None:
+ opener_key = (check_certs, tuple(sorted(newenv.items())))
+ opener = connection_cache.opener_cache.get(opener_key)
+
+ if opener is None:
+ context = self.ssl_context(d)
+ handlers = [FixedHTTPRedirectHandler,
+ HTTPMethodFallback,
+ urllib.request.ProxyHandler(),
+ CacheHTTPHandler(),
+ CacheHTTPSHandler(context=context)]
+ opener = urllib.request.build_opener(*handlers)
+ if connection_cache is not None:
+ connection_cache.opener_cache[opener_key] = opener
try:
parts = urllib.parse.urlparse(ud.url.split(";")[0])
--
2.43.0
next prev parent reply other threads:[~2026-09-28 12:51 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-28 12:50 [PATCH v2 0/2] fetch/wget: Reuse the checkstatus() urllib opener ecordonnier
2026-09-28 12:50 ` [PATCH v2 1/2] fetch/wget: Have checkstatus() capture connection_cache, not fetch ecordonnier
2026-11-16 19:51 ` Etienne Cordonnier
2026-11-17 8:53 ` Paul Barker
2026-09-28 12:50 ` ecordonnier [this message]
2026-11-17 8:58 ` [PATCH v2 2/2] fetch/wget: Reuse the checkstatus() urllib opener Paul Barker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260928125051.2483039-3-ecordonnier@snap.com \
--to=ecordonnier@snap.com \
--cc=bitbake-devel@lists.openembedded.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.