* [PATCH v2] fetch2/wget: reuse cached HTTPS connections
@ 2026-08-12 17:03 Fredrik Svensson (svsvenss)
2026-08-14 13:21 ` [bitbake-devel] " Richard Purdie
2026-08-17 13:48 ` chris.laplante
0 siblings, 2 replies; 11+ messages in thread
From: Fredrik Svensson (svsvenss) @ 2026-08-12 17:03 UTC (permalink / raw)
To: bitbake-devel@lists.openembedded.org
The checkstatus() path reuses cached connections for HTTP requests, but
uses urllib's standard HTTPSHandler for HTTPS. Each HTTPS availability
check therefore creates a new TCP connection and performs a new TLS
handshake.
Add HTTPS equivalents of the existing cached connection and request
handler. Preserve the SSL context selected by BB_CHECK_SSL_CERTS and
extend FetchConnectionCache keys so HTTPS connections are kept separate
from HTTP connections, different certificate policies and CA files, and
different proxy tunnels.
Add a local TLS self-test which proves two HTTPS HEAD requests use one
TCP/TLS connection. The test also verifies that a connection established
with certificate checking disabled is not reused after checking is
enabled.
In an ABBA benchmark against BitBake master, 4,283 remote sstate
availability misses averaged 761.639 seconds without this change and
254.742 seconds with it. This reduced the check by 66.55%, a 2.990x
speedup. The benchmark had no matching mirror objects, so it isolates
availability checks rather than download and unpack time.
Signed-off-by: Fredrik Svensson <svsvenss@cisco.com>
---
Changes in v2:
- Avoid urllib.request.HTTPSHandler's private _check_hostname attribute,
which is no longer present with Python 3.12 and newer.
lib/bb/fetch2/__init__.py | 18 ++++++-----
lib/bb/fetch2/wget.py | 56 +++++++++++++++++++++++++++------
lib/bb/tests/fetch.py | 66 +++++++++++++++++++++++++++++++++++++++
3 files changed, 122 insertions(+), 18 deletions(-)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index a5772f8a..ea007f36 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -2127,26 +2127,28 @@ class FetchConnectionCache(object):
def __init__(self):
self.cache = {}
- def get_connection_name(self, host, port):
- return host + ':' + str(port)
+ def get_connection_name(self, host, port, connection_id=None):
+ if connection_id is None:
+ return host + ':' + str(port)
+ return (host, port, connection_id)
- def add_connection(self, host, port, connection):
- cn = self.get_connection_name(host, port)
+ def add_connection(self, host, port, connection, connection_id=None):
+ cn = self.get_connection_name(host, port, connection_id)
if cn not in self.cache:
self.cache[cn] = connection
- def get_connection(self, host, port):
+ def get_connection(self, host, port, connection_id=None):
connection = None
- cn = self.get_connection_name(host, port)
+ cn = self.get_connection_name(host, port, connection_id)
if cn in self.cache:
connection = self.cache[cn]
return connection
- def remove_connection(self, host, port):
- cn = self.get_connection_name(host, port)
+ def remove_connection(self, host, port, connection_id=None):
+ cn = self.get_connection_name(host, port, connection_id)
if cn in self.cache:
self.cache[cn].close()
del self.cache[cn]
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index 141c2d06..c4534b05 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -151,27 +151,54 @@ class Wget(FetchMethod):
return True
def checkstatus(self, fetch, ud, d, try_again=True):
+ check_certs = self.check_certs(d)
+ newenv = bb.fetch2.get_fetcher_environment(d)
+
class HTTPConnectionCache(http.client.HTTPConnection):
+ def cache_id(self):
+ return None
+
if fetch.connection_cache:
def connect(self):
"""Connect to the host and port specified in __init__."""
- sock = fetch.connection_cache.get_connection(self.host, self.port)
+ sock = fetch.connection_cache.get_connection(
+ self.host, self.port, self.cache_id())
if sock:
self.sock = sock
else:
self.sock = socket.create_connection((self.host, self.port),
self.timeout, self.source_address)
- fetch.connection_cache.add_connection(self.host, self.port, self.sock)
+ fetch.connection_cache.add_connection(
+ self.host, self.port, self.sock, self.cache_id())
if self._tunnel_host:
self._tunnel()
+ class HTTPSConnectionCache(http.client.HTTPSConnection):
+ def cache_id(self):
+ return ("https", check_certs,
+ newenv.get("SSL_CERT_FILE"),
+ self._tunnel_host, self._tunnel_port)
+
+ if fetch.connection_cache:
+ def connect(self):
+ """Reuse an established TLS connection when available."""
+
+ sock = fetch.connection_cache.get_connection(
+ self.host, self.port, self.cache_id())
+ if sock:
+ self.sock = sock
+ else:
+ super().connect()
+ fetch.connection_cache.add_connection(
+ self.host, self.port, self.sock, self.cache_id())
+
class CacheHTTPHandler(urllib.request.HTTPHandler):
def http_open(self, req):
return self.do_open(HTTPConnectionCache, req)
- def do_open(self, http_class, req):
+ def do_open(self, http_class, req, **http_conn_args):
"""Return an addinfourl object for the request, using http_class.
http_class must implement the HTTPConnection API from httplib.
@@ -185,7 +212,7 @@ class Wget(FetchMethod):
if not host:
raise urllib.error.URLError('no host given')
- h = http_class(host, timeout=req.timeout) # will parse host:port
+ h = http_class(host, timeout=req.timeout, **http_conn_args) # will parse host:port
h.set_debuglevel(self._debuglevel)
headers = dict(req.unredirected_hdrs)
@@ -231,7 +258,8 @@ class Wget(FetchMethod):
# If it still fails, we give up, which can happen for bad
# HTTP proxy settings.
if fetch.connection_cache:
- fetch.connection_cache.remove_connection(h.host, h.port)
+ fetch.connection_cache.remove_connection(
+ h.host, h.port, h.cache_id())
h.close()
raise
@@ -265,10 +293,20 @@ class Wget(FetchMethod):
# Close connection when server request it.
if fetch.connection_cache is not None:
if 'Connection' in r.msg and r.msg['Connection'] == 'close':
- fetch.connection_cache.remove_connection(h.host, h.port)
+ fetch.connection_cache.remove_connection(
+ h.host, h.port, h.cache_id())
return resp
+ class CacheHTTPSHandler(CacheHTTPHandler, urllib.request.HTTPSHandler):
+ def __init__(self, debuglevel=0, context=None, check_hostname=None):
+ urllib.request.HTTPSHandler.__init__(self, debuglevel, context,
+ check_hostname)
+
+ def https_open(self, req):
+ return self.do_open(HTTPSConnectionCache, req,
+ context=self._context)
+
class HTTPMethodFallback(urllib.request.BaseHandler):
"""
Fallback to GET if HEAD is not allowed (405 HTTP error)
@@ -370,12 +408,10 @@ class Wget(FetchMethod):
# Avoid tramping the environment too much by using bb.utils.environment
# to scope the changes to the build_opener request, which is when the
# environment lookups happen.
- newenv = bb.fetch2.get_fetcher_environment(d)
-
with bb.utils.environment(**newenv):
import ssl
- if self.check_certs(d):
+ if check_certs:
context = ssl.create_default_context()
else:
context = ssl._create_unverified_context()
@@ -384,7 +420,7 @@ class Wget(FetchMethod):
HTTPMethodFallback,
urllib.request.ProxyHandler(),
CacheHTTPHandler(),
- urllib.request.HTTPSHandler(context=context)]
+ CacheHTTPSHandler(context=context)]
opener = urllib.request.build_opener(*handlers)
try:
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index cd50c37a..fd064743 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1781,6 +1781,72 @@ class FetchCheckStatusTest(FetcherTest):
connection_cache.close_connections()
+ @unittest.skipUnless(shutil.which("openssl"), "openssl not installed")
+ def test_wget_checkstatus_https_connection_cache(self):
+ import ssl
+ from socketserver import ThreadingMixIn
+ from bb.fetch2 import FetchConnectionCache
+
+ class HTTPSRequestHandler(http.server.BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def do_HEAD(self):
+ self.send_response(200)
+ self.send_header("Content-Length", "0")
+ self.end_headers()
+
+ def log_message(self, format_str, *args):
+ pass
+
+ class HTTPSServer(ThreadingMixIn, http.server.HTTPServer):
+ daemon_threads = True
+
+ def __init__(self, *args, **kwargs):
+ self.connection_count = 0
+ super().__init__(*args, **kwargs)
+
+ def get_request(self):
+ request, client_address = super().get_request()
+ self.connection_count += 1
+ return request, client_address
+
+ certificate = os.path.join(self.tempdir, "certificate.pem")
+ private_key = os.path.join(self.tempdir, "private-key.pem")
+ subprocess.check_call(
+ ["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes",
+ "-keyout", private_key, "-out", certificate, "-days", "1",
+ "-subj", "/CN=127.0.0.1"],
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+
+ server = HTTPSServer(("127.0.0.1", 0), HTTPSRequestHandler)
+ context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+ context.load_cert_chain(certificate, private_key)
+ server.socket = context.wrap_socket(server.socket, server_side=True)
+ server_thread = threading.Thread(target=server.serve_forever)
+ server_thread.daemon = True
+ server_thread.start()
+
+ connection_cache = FetchConnectionCache()
+ try:
+ url = "https://127.0.0.1:%s/test" % server.server_port
+ self.d.setVar("BB_CHECK_SSL_CERTS", "0")
+ fetch = bb.fetch2.Fetch([url], self.d,
+ connection_cache=connection_cache)
+ ud = fetch.ud[url]
+ self.assertTrue(ud.method.checkstatus(fetch, ud, self.d))
+ self.assertTrue(ud.method.checkstatus(fetch, ud, self.d))
+ self.assertEqual(server.connection_count, 1)
+
+ # A connection established without certificate checks must not be
+ # reused after certificate checking is enabled.
+ self.d.setVar("BB_CHECK_SSL_CERTS", "1")
+ self.assertFalse(ud.method.checkstatus(fetch, ud, self.d))
+ finally:
+ connection_cache.close_connections()
+ server.shutdown()
+ server_thread.join()
+ server.server_close()
+
def test_wget_checkstatus_same_origin_redirect_keeps_auth(self):
server = self._start_checkstatus_server()
server.redirect_url = "http://127.0.0.1:%s/b" % server.server_port
--
2.55.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-12 17:03 [PATCH v2] fetch2/wget: reuse cached HTTPS connections Fredrik Svensson (svsvenss)
@ 2026-08-14 13:21 ` Richard Purdie
2026-08-14 20:12 ` Fredrik Svensson (svsvenss)
2026-08-17 13:48 ` chris.laplante
1 sibling, 1 reply; 11+ messages in thread
From: Richard Purdie @ 2026-08-14 13:21 UTC (permalink / raw)
To: svsvenss, bitbake-devel@lists.openembedded.org; +Cc: Mathieu Dubois-Briand
Hi,
On Wed, 2026-08-12 at 17:03 +0000, Fredrik Svensson via lists.openembedded.org wrote:
> The checkstatus() path reuses cached connections for HTTP requests, but
> uses urllib's standard HTTPSHandler for HTTPS. Each HTTPS availability
> check therefore creates a new TCP connection and performs a new TLS
> handshake.
>
> Add HTTPS equivalents of the existing cached connection and request
> handler. Preserve the SSL context selected by BB_CHECK_SSL_CERTS and
> extend FetchConnectionCache keys so HTTPS connections are kept separate
> from HTTP connections, different certificate policies and CA files, and
> different proxy tunnels.
>
> Add a local TLS self-test which proves two HTTPS HEAD requests use one
> TCP/TLS connection. The test also verifies that a connection established
> with certificate checking disabled is not reused after checking is
> enabled.
>
> In an ABBA benchmark against BitBake master, 4,283 remote sstate
> availability misses averaged 761.639 seconds without this change and
> 254.742 seconds with it. This reduced the check by 66.55%, a 2.990x
> speedup. The benchmark had no matching mirror objects, so it isolates
> availability checks rather than download and unpack time.
>
> Signed-off-by: Fredrik Svensson <svsvenss@cisco.com>
Thanks for the patch, this looked good to me and we did merge it. We then started seeing:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4378/steps/15/logs/stdio
which is probably worker specific and occuring on alma8 (which we do
use buildtools tarball on). Have you any idea why that might be
breaking?
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-14 13:21 ` [bitbake-devel] " Richard Purdie
@ 2026-08-14 20:12 ` Fredrik Svensson (svsvenss)
2026-08-14 21:24 ` Richard Purdie
0 siblings, 1 reply; 11+ messages in thread
From: Fredrik Svensson (svsvenss) @ 2026-08-14 20:12 UTC (permalink / raw)
To: Richard Purdie, bitbake-devel@lists.openembedded.org
Cc: Mathieu Dubois-Briand
[-- Attachment #1: Type: text/plain, Size: 2707 bytes --]
The failure happens while the new self-test is generating its temporary certificate, before it exercises the HTTPS connection cache. Unfortunately, the test currently redirects both stdout and stderr to /dev/null, so the log only contains the exit status.
I reproduced the command in an AlmaLinux 8 container using the same Yocto 5.1 buildtools, where it succeeds. This therefore looks specific to the alma8-vk-2worker, possibly its crypto/FIPS configuration or local state.
I will prepare a small follow-up patch which captures the OpenSSL output and includes it in the log. A rerun with that patch should expose the actual reason.
________________________________
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Sent: Friday, August 14, 2026 3:21 PM
To: Fredrik Svensson (svsvenss) <svsvenss@cisco.com>; bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Cc: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Subject: Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
Hi,
On Wed, 2026-08-12 at 17:03 +0000, Fredrik Svensson via lists.openembedded.org wrote:
> The checkstatus() path reuses cached connections for HTTP requests, but
> uses urllib's standard HTTPSHandler for HTTPS. Each HTTPS availability
> check therefore creates a new TCP connection and performs a new TLS
> handshake.
>
> Add HTTPS equivalents of the existing cached connection and request
> handler. Preserve the SSL context selected by BB_CHECK_SSL_CERTS and
> extend FetchConnectionCache keys so HTTPS connections are kept separate
> from HTTP connections, different certificate policies and CA files, and
> different proxy tunnels.
>
> Add a local TLS self-test which proves two HTTPS HEAD requests use one
> TCP/TLS connection. The test also verifies that a connection established
> with certificate checking disabled is not reused after checking is
> enabled.
>
> In an ABBA benchmark against BitBake master, 4,283 remote sstate
> availability misses averaged 761.639 seconds without this change and
> 254.742 seconds with it. This reduced the check by 66.55%, a 2.990x
> speedup. The benchmark had no matching mirror objects, so it isolates
> availability checks rather than download and unpack time.
>
> Signed-off-by: Fredrik Svensson <svsvenss@cisco.com>
Thanks for the patch, this looked good to me and we did merge it. We then started seeing:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4378/steps/15/logs/stdio
which is probably worker specific and occuring on alma8 (which we do
use buildtools tarball on). Have you any idea why that might be
breaking?
Cheers,
Richard
[-- Attachment #2: Type: text/html, Size: 4591 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-14 20:12 ` Fredrik Svensson (svsvenss)
@ 2026-08-14 21:24 ` Richard Purdie
2026-08-17 6:55 ` Fredrik Svensson (svsvenss)
0 siblings, 1 reply; 11+ messages in thread
From: Richard Purdie @ 2026-08-14 21:24 UTC (permalink / raw)
To: svsvenss, bitbake-devel@lists.openembedded.org; +Cc: Mathieu Dubois-Briand
On Fri, 2026-08-14 at 20:12 +0000, Fredrik Svensson via lists.openembedded.org wrote:
> The failure happens while the new self-test is generating its
> temporary certificate, before it exercises the HTTPS connection
> cache. Unfortunately, the test currently redirects both stdout and
> stderr to /dev/null, so the log only contains the exit status.
>
> I reproduced the command in an AlmaLinux 8 container using the same
> Yocto 5.1 buildtools, where it succeeds. This therefore looks
> specific to the alma8-vk-2worker, possibly its crypto/FIPS
> configuration or local state.
>
> I will prepare a small follow-up patch which captures the OpenSSL
> output and includes it in the log. A rerun with that patch should
> expose the actual reason.
Thanks, I applied it and the result was:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4383/steps/15/logs/stdio
which suggests buildtools isn't working quite right. What is odd is
that if I ssh in and run that bitbake-selftest command on the same
build directory, it works and doesn't show the error. I'm not quite
sure what is going on here...
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-14 21:24 ` Richard Purdie
@ 2026-08-17 6:55 ` Fredrik Svensson (svsvenss)
2026-08-17 14:07 ` Richard Purdie
0 siblings, 1 reply; 11+ messages in thread
From: Fredrik Svensson (svsvenss) @ 2026-08-17 6:55 UTC (permalink / raw)
To: Richard Purdie, bitbake-devel@lists.openembedded.org
Cc: Mathieu Dubois-Briand
[-- Attachment #1: Type: text/plain, Size: 2610 bytes --]
Thanks, the new output makes the problem clearer.
The buildtools OpenSSL is trying to load its configuration from the
original SDK build path:
/usr/local/oe-sdk-hardcoded-buildpath/.../openssl.cnf
It looks like enable_tools_tarball() adds the buildtools binaries to
PATH, but does not source the scripts under environment-setup.d. The
OPENSSL_CONF value is set by environment-setup.d/openssl.sh, so
OpenSSL falls back to its compiled-in path instead.
I reproduced the same error with the 5.1 buildtools tarball when using
its OpenSSL without sourcing the complete environment. It works when
the environment is sourced normally.
I think the proper fix is for enable_tools_tarball() to import the complete
sourced environment, including the environment-setup.d scripts.
That would also explain why running the test manually over SSH works. I
think this is an issue with how yocto-autobuilder-helper imports the
buildtools environment, rather than with the BitBake HTTPS cache change.
Kind regards
Fredrik
________________________________
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Sent: Friday, August 14, 2026 11:24 PM
To: Fredrik Svensson (svsvenss) <svsvenss@cisco.com>; bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Cc: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Subject: Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
On Fri, 2026-08-14 at 20:12 +0000, Fredrik Svensson via lists.openembedded.org wrote:
> The failure happens while the new self-test is generating its
> temporary certificate, before it exercises the HTTPS connection
> cache. Unfortunately, the test currently redirects both stdout and
> stderr to /dev/null, so the log only contains the exit status.
>
> I reproduced the command in an AlmaLinux 8 container using the same
> Yocto 5.1 buildtools, where it succeeds. This therefore looks
> specific to the alma8-vk-2worker, possibly its crypto/FIPS
> configuration or local state.
>
> I will prepare a small follow-up patch which captures the OpenSSL
> output and includes it in the log. A rerun with that patch should
> expose the actual reason.
Thanks, I applied it and the result was:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4383/steps/15/logs/stdio
which suggests buildtools isn't working quite right. What is odd is
that if I ssh in and run that bitbake-selftest command on the same
build directory, it works and doesn't show the error. I'm not quite
sure what is going on here...
Cheers,
Richard
[-- Attachment #2: Type: text/html, Size: 8588 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-12 17:03 [PATCH v2] fetch2/wget: reuse cached HTTPS connections Fredrik Svensson (svsvenss)
2026-08-14 13:21 ` [bitbake-devel] " Richard Purdie
@ 2026-08-17 13:48 ` chris.laplante
2026-08-17 14:14 ` Fredrik Svensson (svsvenss)
1 sibling, 1 reply; 11+ messages in thread
From: chris.laplante @ 2026-08-17 13:48 UTC (permalink / raw)
To: svsvenss@cisco.com, bitbake-devel@lists.openembedded.org
Hi Fredrik,
> The checkstatus() path reuses cached connections for HTTP requests, but uses
> urllib's standard HTTPSHandler for HTTPS. Each HTTPS availability check
> therefore creates a new TCP connection and performs a new TLS handshake.
>
> Add HTTPS equivalents of the existing cached connection and request handler.
> Preserve the SSL context selected by BB_CHECK_SSL_CERTS and extend
> FetchConnectionCache keys so HTTPS connections are kept separate from HTTP
> connections, different certificate policies and CA files, and different proxy
> tunnels.
>
> Add a local TLS self-test which proves two HTTPS HEAD requests use one
> TCP/TLS connection. The test also verifies that a connection established with
> certificate checking disabled is not reused after checking is enabled.
>
> In an ABBA benchmark against BitBake master, 4,283 remote sstate availability
> misses averaged 761.639 seconds without this change and
> 254.742 seconds with it. This reduced the check by 66.55%, a 2.990x speedup.
> The benchmark had no matching mirror objects, so it isolates availability checks
> rather than download and unpack time.
Funny enough, just last week I was also poking around in this area using viztracer to do some profiling. I was preparing to send a patch that just cached the SSL context creation, which on my system costs 100-200ms each time (since it has to load the certificate store from disk).
Thanks,
Chris
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-17 6:55 ` Fredrik Svensson (svsvenss)
@ 2026-08-17 14:07 ` Richard Purdie
0 siblings, 0 replies; 11+ messages in thread
From: Richard Purdie @ 2026-08-17 14:07 UTC (permalink / raw)
To: Fredrik Svensson (svsvenss), bitbake-devel@lists.openembedded.org
Cc: Mathieu Dubois-Briand
On Mon, 2026-08-17 at 06:55 +0000, Fredrik Svensson (svsvenss) wrote:
> Thanks, the new output makes the problem clearer.
>
>
> The buildtools OpenSSL is trying to load its configuration from the
> original SDK build path:
>
>
> /usr/local/oe-sdk-hardcoded-buildpath/.../openssl.cnf
>
>
> It looks like enable_tools_tarball() adds the buildtools binaries to
> PATH, but does not source the scripts under environment-setup.d. The
> OPENSSL_CONF value is set by environment-setup.d/openssl.sh, so
> OpenSSL falls back to its compiled-in path instead.
>
>
> I reproduced the same error with the 5.1 buildtools tarball when
> using its OpenSSL without sourcing the complete environment. It works
> when the environment is sourced normally.
>
>
> I think the proper fix is for enable_tools_tarball() to import the
> complete sourced environment, including the environment-setup.d
> scripts.
>
>
> That would also explain why running the test manually over SSH works.
> I think this is an issue with how yocto-autobuilder-helper imports
> the buildtools environment, rather than with the BitBake HTTPS cache
> change.
Agreed, I was coming to the same conclusion. The issue is that code in
the autobuilder helper... :/
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-17 13:48 ` chris.laplante
@ 2026-08-17 14:14 ` Fredrik Svensson (svsvenss)
2026-08-17 14:17 ` chris.laplante
0 siblings, 1 reply; 11+ messages in thread
From: Fredrik Svensson (svsvenss) @ 2026-08-17 14:14 UTC (permalink / raw)
To: chris.laplante@agilent.com, bitbake-devel@lists.openembedded.org
[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]
Hi Chris,
Caching the SSL context should provide an additional improvement, especially if
context creation takes 100-200ms on your system.
Please do send the patch; I think the two changes should work well together.
Cheers
Fredrik
________________________________
From: chris.laplante@agilent.com <chris.laplante@agilent.com>
Sent: Monday, August 17, 2026 3:48 PM
To: Fredrik Svensson (svsvenss) <svsvenss@cisco.com>; bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Subject: RE: [PATCH v2] fetch2/wget: reuse cached HTTPS connections
Hi Fredrik,
> The checkstatus() path reuses cached connections for HTTP requests, but uses
> urllib's standard HTTPSHandler for HTTPS. Each HTTPS availability check
> therefore creates a new TCP connection and performs a new TLS handshake.
>
> Add HTTPS equivalents of the existing cached connection and request handler.
> Preserve the SSL context selected by BB_CHECK_SSL_CERTS and extend
> FetchConnectionCache keys so HTTPS connections are kept separate from HTTP
> connections, different certificate policies and CA files, and different proxy
> tunnels.
>
> Add a local TLS self-test which proves two HTTPS HEAD requests use one
> TCP/TLS connection. The test also verifies that a connection established with
> certificate checking disabled is not reused after checking is enabled.
>
> In an ABBA benchmark against BitBake master, 4,283 remote sstate availability
> misses averaged 761.639 seconds without this change and
> 254.742 seconds with it. This reduced the check by 66.55%, a 2.990x speedup.
> The benchmark had no matching mirror objects, so it isolates availability checks
> rather than download and unpack time.
Funny enough, just last week I was also poking around in this area using viztracer to do some profiling. I was preparing to send a patch that just cached the SSL context creation, which on my system costs 100-200ms each time (since it has to load the certificate store from disk).
Thanks,
Chris
[-- Attachment #2: Type: text/html, Size: 4242 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-17 14:14 ` Fredrik Svensson (svsvenss)
@ 2026-08-17 14:17 ` chris.laplante
2026-08-17 19:34 ` [bitbake-devel] " Richard Purdie
0 siblings, 1 reply; 11+ messages in thread
From: chris.laplante @ 2026-08-17 14:17 UTC (permalink / raw)
To: Fredrik Svensson (svsvenss), bitbake-devel@lists.openembedded.org
Hi Fredik,
> Hi Chris,
> Caching the SSL context should provide an additional improvement, especially if
> context creation takes 100-200ms on your system.
>
> Please do send the patch; I think the two changes should work well together.
>
> Cheers
> Fredrik
> ________________________________________
> From: mailto:chris.laplante@agilent.com <mailto:chris.laplante@agilent.com>
> Sent: Monday, August 17, 2026 3:48 PM
> To: Fredrik Svensson (svsvenss) <mailto:svsvenss@cisco.com>; mailto:bitbake-
> devel@lists.openembedded.org <mailto:bitbake-
> devel@lists.openembedded.org>
> Subject: RE: [PATCH v2] fetch2/wget: reuse cached HTTPS connections
>
> ...
>
> Funny enough, just last week I was also poking around in this area using
> viztracer to do some profiling. I was preparing to send a patch that just cached
> the SSL context creation, which on my system costs 100-200ms each time (since
> it has to load the certificate store from disk).
Will do. I think I'll hold off a bit until your changes land in master-next though, to avoid merge conflicts.
Thanks,
Chris
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-17 14:17 ` chris.laplante
@ 2026-08-17 19:34 ` Richard Purdie
2026-08-17 19:36 ` chris.laplante
0 siblings, 1 reply; 11+ messages in thread
From: Richard Purdie @ 2026-08-17 19:34 UTC (permalink / raw)
To: chris.laplante, Fredrik Svensson (svsvenss),
bitbake-devel@lists.openembedded.org
On Mon, 2026-08-17 at 14:17 +0000, Chris Laplante via lists.openembedded.org wrote:
> > Funny enough, just last week I was also poking around in this area using
> > viztracer to do some profiling. I was preparing to send a patch that just cached
> > the SSL context creation, which on my system costs 100-200ms each time (since
> > it has to load the certificate store from disk).
>
> Will do. I think I'll hold off a bit until your changes land in
> master-next though, to avoid merge conflicts.
I think they're already in master?
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [bitbake-devel] [PATCH v2] fetch2/wget: reuse cached HTTPS connections
2026-08-17 19:34 ` [bitbake-devel] " Richard Purdie
@ 2026-08-17 19:36 ` chris.laplante
0 siblings, 0 replies; 11+ messages in thread
From: chris.laplante @ 2026-08-17 19:36 UTC (permalink / raw)
To: Richard Purdie, Fredrik Svensson (svsvenss),
bitbake-devel@lists.openembedded.org
> > > Funny enough, just last week I was also poking around in this area
> > > using viztracer to do some profiling. I was preparing to send a
> > > patch that just cached the SSL context creation, which on my system
> > > costs 100-200ms each time (since it has to load the certificate store from
> disk).
> >
> > Will do. I think I'll hold off a bit until your changes land in
> > master-next though, to avoid merge conflicts.
>
> I think they're already in master?
My mistake, I missed that. Will work on them now. Thanks!
Chris
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2026-08-17 19:36 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-12 17:03 [PATCH v2] fetch2/wget: reuse cached HTTPS connections Fredrik Svensson (svsvenss)
2026-08-14 13:21 ` [bitbake-devel] " Richard Purdie
2026-08-14 20:12 ` Fredrik Svensson (svsvenss)
2026-08-14 21:24 ` Richard Purdie
2026-08-17 6:55 ` Fredrik Svensson (svsvenss)
2026-08-17 14:07 ` Richard Purdie
2026-08-17 13:48 ` chris.laplante
2026-08-17 14:14 ` Fredrik Svensson (svsvenss)
2026-08-17 14:17 ` chris.laplante
2026-08-17 19:34 ` [bitbake-devel] " Richard Purdie
2026-08-17 19:36 ` chris.laplante
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.