* [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
@ 2026-06-15 12:18 Jaipaul Cheernam
2026-06-15 13:31 ` [bitbake-devel] " Alexander Kanavin
` (5 more replies)
0 siblings, 6 replies; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-15 12:18 UTC (permalink / raw)
To: bitbake-devel; +Cc: Jaipaul Cheernam
Allow distro-specific buildtools to be configured with a custom URL,
filename, and SDK name via an install-buildtools section in
config-upstream.json. Expose the same options as CLI arguments: --url,
--filename, --sdk-name, --no-check. CLI arguments take precedence over
config values.
This requires the --sdk-name option to be available in
oe-scripts/install-buildtools from openembedded-core.
AI-Generated: Kiro with Claude Opus 4.6
Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
---
bin/bitbake-setup | 49 +++++++++++-
.../bitbake-user-manual-environment-setup.rst | 33 ++++++++
lib/bb/tests/setup.py | 80 ++++++++++++++++++-
setup-schema/bitbake-setup.schema.json | 30 +++++++
4 files changed, 188 insertions(+), 4 deletions(-)
diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index fe3b6b0e6..343d428e8 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -1052,10 +1052,49 @@ def install_buildtools(top_dir, settings, args, d):
return
shutil.rmtree(buildtools_install_dir)
- install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+ install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+
+ buildtools_config = {}
+ upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
+ try:
+ with open(upstream_config_path) as f:
+ upstream_config = json.load(f)
+ buildtools_config = upstream_config.get('data', {}).get('bitbake-setup', {}).get('install-buildtools', {})
+ except FileNotFoundError:
+ pass
+ except json.JSONDecodeError as e:
+ logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
+ sys.exit(1)
+
+ url = args.url if args.url is not None else buildtools_config.get('url')
+ filename = args.filename if args.filename is not None else buildtools_config.get('filename')
+ sdk_name = args.sdk_name if args.sdk_name is not None else buildtools_config.get('sdk-name')
+
+ for name, value in (("url", url), ("filename", filename), ("sdk-name", sdk_name)):
+ if value == "":
+ logger.error("%s is empty" % name)
+ sys.exit(1)
+ no_check = args.no_check if args.no_check is not None else buildtools_config.get('no-check', False)
+
+ if (url is None) != (filename is None):
+ logger.error("url and filename must both be set (from CLI or config) or both omitted")
+ sys.exit(1)
+
+ cmd = [
+ install_buildtools_script,
+ '-d', buildtools_install_dir,
+ '--downloads-directory', buildtools_download_dir,
+ ]
+ if url and filename:
+ cmd += ['--url', url, '--filename', filename]
+ if sdk_name:
+ cmd += ['--sdk-name', sdk_name]
+ if no_check:
+ cmd.append('--no-check')
+
logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
- subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
+ subprocess.check_call(cmd)
def create_siteconf(top_dir, non_interactive, settings):
siteconfpath = os.path.join(top_dir, 'site.conf')
@@ -1285,6 +1324,12 @@ def main():
parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
add_setup_dir_arg(parser_install_buildtools)
parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
+ parser_install_buildtools.add_argument('--url', action='store', help='URL from where to fetch the buildtools SDK installer, not including filename. Requires --filename. Overrides config value.')
+ parser_install_buildtools.add_argument('--filename', action='store', help='Filename for the buildtools SDK installer. Requires --url. Overrides config value.')
+ parser_install_buildtools.add_argument('--sdk-name', action='store', default=None, help='SDK name used in the environment setup script (e.g. customsdk). Overrides config value.')
+ check_group = parser_install_buildtools.add_mutually_exclusive_group()
+ check_group.add_argument('--no-check', action='store_const', const=True, default=None, dest='no_check', help='Disable checksum validation. Overrides config value.')
+ check_group.add_argument('--check', action='store_const', const=False, dest='no_check', help='Enable checksum validation. Overrides config no-check: true.')
parser_install_buildtools.set_defaults(func=install_buildtools)
parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
index aa546c30b..931fda798 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -660,6 +660,39 @@ In addition, the command can take the following arguments:
- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
required if :term:`BBPATH` is already configured.
+- ``--url``: URL from where to fetch the buildtools SDK installer, not
+ including filename. Requires ``--filename``. Overrides the value from the
+ configuration file.
+
+- ``--filename``: filename of the buildtools installer. Requires ``--url``.
+ Overrides the value from the configuration file.
+
+- ``--sdk-name``: SDK name used in the environment setup script installed
+ by the buildtools tarball (e.g. ``customsdk`` for
+ ``environment-setup-<arch>-customsdk-linux``). Defaults to ``pokysdk``.
+ Overrides the value from the configuration file.
+
+- ``--no-check``: disable checksum validation when the server does not
+ provide a ``.sha256sum`` file alongside the installer. Checksum
+ validation is enabled by default.
+
+The ``url``, ``filename``, ``sdk-name`` and ``no-check`` options can also be
+set in the :term:`Configuration File` under the ``bitbake-setup`` section::
+
+ "bitbake-setup": {
+ "install-buildtools": {
+ "url": "https://example.com/buildtools",
+ "filename": "x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
+ "sdk-name": "customsdk",
+ "no-check": true
+ },
+ "configurations": [ ... ]
+ }
+
+CLI arguments take precedence over values in the configuration file.
+Note that setting ``no-check`` to ``false`` in the configuration file has no
+effect; omit the key to use the default (enabled).
+
.. _ref-bbsetup-command-settings:
``bitbake-setup settings``
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index 5592e8196..f14da5a60 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -11,6 +11,7 @@ import glob
import hashlib
import json
import os
+import shutil
import stat
from bb.tests.support.httpserver import HTTPService
@@ -61,13 +62,22 @@ import getopt
import sys
import os
-opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
+opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "url=", "filename=", "sdk-name=", "no-check"])
+installdir = None
for option, value in opts:
if option == '-d':
installdir = value
+ elif option == '--url':
+ print("install-buildtools url={}".format(value))
+ elif option == '--filename':
+ print("install-buildtools filename={}".format(value))
+ elif option == '--sdk-name':
+ print("install-buildtools sdk-name={}".format(value))
+ elif option == '--no-check':
+ print("install-buildtools no-check")
print("Buildtools installed into {}".format(installdir))
-os.makedirs(installdir)
+os.makedirs(installdir, exist_ok=True)
"""
self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
@@ -644,6 +654,72 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
def _count_layer_backups(self, layers_path):
return len([f for f in os.listdir(layers_path) if 'backup' in f])
+ def test_install_buildtools_options(self):
+ """Test that url, filename, sdk-name and no-check are passed through to install-buildtools"""
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ self.runbbsetup("settings set default registry 'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath))
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
+ self.runbbsetup("init --non-interactive test-config-bt gadget")
+ setuppath = self.get_setup_path('test-config-bt', 'gadget')
+
+ # test CLI options are passed through
+ out = self.runbbsetup("install-buildtools --setup-dir {} "
+ "--url https://example.com/buildtools "
+ "--filename x86_64-buildtools.sh "
+ "--sdk-name customsdk "
+ "--no-check".format(setuppath))
+ self.assertIn("install-buildtools url=https://example.com/buildtools", out[0])
+ self.assertIn("install-buildtools filename=x86_64-buildtools.sh", out[0])
+ self.assertIn("install-buildtools sdk-name=customsdk", out[0])
+ self.assertIn("install-buildtools no-check", out[0])
+
+ # test config options are passed through
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+ config_upstream = json.load(f)
+ config_upstream['data']['bitbake-setup']['install-buildtools'] = {
+ 'url': 'https://example.com/buildtools',
+ 'filename': 'x86_64-buildtools-from-config.sh',
+ 'sdk-name': 'customsdk',
+ 'no-check': True
+ }
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+ json.dump(config_upstream, f)
+ out = self.runbbsetup("install-buildtools --setup-dir {}".format(setuppath))
+ self.assertIn("install-buildtools url=https://example.com/buildtools", out[0])
+ self.assertIn("install-buildtools filename=x86_64-buildtools-from-config.sh", out[0])
+ self.assertIn("install-buildtools sdk-name=customsdk", out[0])
+ self.assertIn("install-buildtools no-check", out[0])
+
+ # test CLI overrides config — pass --url/--filename on CLI to avoid implicit dependency on case 2 config state
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ out = self.runbbsetup("install-buildtools --setup-dir {} "
+ "--url https://example.com/buildtools "
+ "--filename x86_64-buildtools.sh "
+ "--sdk-name overridden".format(setuppath))
+ self.assertIn("install-buildtools sdk-name=overridden", out[0])
+ self.assertIn("install-buildtools url=https://example.com/buildtools", out[0])
+ self.assertIn("install-buildtools filename=x86_64-buildtools.sh", out[0])
+
+ # test --check overrides config no-check: true
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ out = self.runbbsetup("install-buildtools --setup-dir {} --check".format(setuppath))
+ self.assertNotIn("install-buildtools no-check", out[0])
+
+ # test --url without --filename is rejected
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+ config_upstream = json.load(f)
+ config_upstream['data']['bitbake-setup'].pop('install-buildtools', None)
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+ json.dump(config_upstream, f)
+ # --force used instead of shutil.rmtree since buildtools dir exists from prior subcase
+ with self.assertRaises(bb.process.ExecutionError):
+ self.runbbsetup("install-buildtools --setup-dir {} --force --url https://example.com/buildtools".format(setuppath))
+
def test_update_rebase_conflicts_strategy(self):
"""Test the --rebase-conflicts-strategy option for the update command.
diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
index 99f47f73d..2a967c96e 100644
--- a/setup-schema/bitbake-setup.schema.json
+++ b/setup-schema/bitbake-setup.schema.json
@@ -141,6 +141,36 @@
},
"additionalProperties": false
}
+ },
+ "install-buildtools": {
+ "type": "object",
+ "description": "Optional settings passed through to oe-scripts/install-buildtools",
+ "properties": {
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "description": "URL from where to fetch the buildtools SDK installer, not including filename. Requires 'filename'."
+ },
+ "filename": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Filename for the buildtools SDK installer. Requires 'url'."
+ },
+ "sdk-name": {
+ "type": "string",
+ "minLength": 1,
+ "description": "SDK name in the environment setup script (e.g. 'customsdk'). Defaults to 'pokysdk'."
+ },
+ "no-check": {
+ "type": "boolean",
+ "description": "Set to true to disable checksum validation when the server does not provide a .sha256sum file. no-check: false is equivalent to omitting the key."
+ }
+ },
+ "dependencies": {
+ "url": ["filename"],
+ "filename": ["url"]
+ },
+ "additionalProperties": false
}
},
"additionalProperties": false
--
2.34.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-15 12:18 [PATCH] bitbake-setup: pass install-buildtools options from config and CLI Jaipaul Cheernam
@ 2026-06-15 13:31 ` Alexander Kanavin
2026-06-15 14:59 ` Jaipaul Cheernam
2026-06-16 10:06 ` Richard Purdie
` (4 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Alexander Kanavin @ 2026-06-15 13:31 UTC (permalink / raw)
To: jaipaul.cheernam; +Cc: bitbake-devel
On Mon, 15 Jun 2026 at 14:19, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
>
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.
Can you please describe the use case? Presumably this is useful in
your environment, I'd like to understand the specifics of what you put
in the custom buildtools archive and why it has to be there and not
built via recipes.
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-15 13:31 ` [bitbake-devel] " Alexander Kanavin
@ 2026-06-15 14:59 ` Jaipaul Cheernam
2026-06-15 15:38 ` Alexander Kanavin
0 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-15 14:59 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: bitbake-devel@lists.openembedded.org
[-- Attachment #1: Type: text/plain, Size: 2266 bytes --]
Hi Alex,
Any downstream distro that builds its own extended buildtools tarball under a non-Poky distro name hits a FileNotFoundError because install-buildtools
hardcodes pokysdk when looking for environment-setup-<arch>-pokysdk-linux. The --sdk-name option fixes that hardcoded assumption.
The archive itself contains the same tools as upstream extended buildtools — it can't be provided via recipes because buildtools exist specifically to
bootstrap the host environment before BitBake can run.
The --url and --filename options are needed for projects that host their buildtools on an internal server rather than yoctoproject.org, and want to pin a
specific version across the team without requiring manual arguments on every invocation.
The workaround today is to pass arguments manually or modify wrapper scripts, which is probably why this hasn't surfaced as a bug report before.
The --sdk-name option itself is introduced in a companion patch to oe-scripts/install-buildtools in openembedded-core:
https://lists.openembedded.org/g/openembedded-core/topic/patch_install_buildtools/119814299
Thanks,
Jaipaul Cheernam
From: Alexander Kanavin <alex.kanavin@gmail.com>
Date: Monday, 15 June 2026 at 15:32
To: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
Cc: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Subject: Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
On Mon, 15 Jun 2026 at 14:19, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
>
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.
Can you please describe the use case? Presumably this is useful in
your environment, I'd like to understand the specifics of what you put
in the custom buildtools archive and why it has to be there and not
built via recipes.
Alex
[-- Attachment #2: Type: text/html, Size: 5696 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-15 14:59 ` Jaipaul Cheernam
@ 2026-06-15 15:38 ` Alexander Kanavin
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Kanavin @ 2026-06-15 15:38 UTC (permalink / raw)
To: Jaipaul Cheernam; +Cc: bitbake-devel@lists.openembedded.org
On Mon, 15 Jun 2026 at 16:59, Jaipaul Cheernam
<jaipaul.cheernam@est.tech> wrote:
> The archive itself contains the same tools as upstream extended buildtools — it can't be provided via recipes because buildtools exist specifically to
> bootstrap the host environment before BitBake can run.
>
> The --url and --filename options are needed for projects that host their buildtools on an internal server rather than yoctoproject.org, and want to pin a
> specific version across the team without requiring manual arguments on every invocation.
>
> The workaround today is to pass arguments manually or modify wrapper scripts, which is probably why this hasn't surfaced as a bug report before.
>
> The --sdk-name option itself is introduced in a companion patch to oe-scripts/install-buildtools in openembedded-core:
> https://lists.openembedded.org/g/openembedded-core/topic/patch_install_buildtools/119814299
Thanks. I'm broadly in favor of this, but I don't know what the
bitbake patch reviewers will say, and especially RP :) Buildtools can
be used to do isolated, reproducible 'container builds' without actual
containers, and I like that.
I wonder if the archive should be downloaded by bitbake-setup itself
using bitbake fetchers (similar to how layers are obtained), and then
given to install-buildtools as a local file, or even executed
directly. There are benefits to this: it will be preserved in the
download cache together with the layers and upstream components (which
helps if upstream server isn't available or altogether disappears),
and checksummed against sha256sum provided in the config (which adds a
reproducibility check, and protection against supply chain attacks,
similar to tarball recipes). What you think?
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-15 12:18 [PATCH] bitbake-setup: pass install-buildtools options from config and CLI Jaipaul Cheernam
2026-06-15 13:31 ` [bitbake-devel] " Alexander Kanavin
@ 2026-06-16 10:06 ` Richard Purdie
2026-06-16 14:18 ` Jaipaul Cheernam
2026-06-26 13:43 ` [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download Jaipaul Cheernam
` (3 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Richard Purdie @ 2026-06-16 10:06 UTC (permalink / raw)
To: jaipaul.cheernam, bitbake-devel
On Mon, 2026-06-15 at 14:18 +0200, Jaipaul Cheernam via lists.openembedded.org wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
>
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.
I replied to the oe-core patch, lets just remove the need to have --
sdk-name, I don't think we need that. I also don't understand why you
want/need the --no-check option, we should be able to tell if we should
be checking the checksum or not? If that means fixing the install
script, so be it.
Cheers,
Richard
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-16 10:06 ` Richard Purdie
@ 2026-06-16 14:18 ` Jaipaul Cheernam
2026-06-16 14:26 ` Alexander Kanavin
0 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-16 14:18 UTC (permalink / raw)
To: Richard Purdie, bitbake-devel@lists.openembedded.org
[-- Attachment #1: Type: text/plain, Size: 2098 bytes --]
Hi Richard,
Agreed on removing --sdk-name and I have sent Patch v2 for review in oe-core.
For --no-check — I'm fine removing it and having the install-buildtools auto-detect, but we need to decide on the failure mode: if the checksum file can't be fetched
(404 or network error), should we:
(a) warn and continue the install without verification, or
(b) hard fail and refuse to install?
If (a), we're effectively auto---no-check on failure, which may silently skip validation on transient network issues. If (b), users with custom/local
URLs that don't host checksum files would have no way to install without us re-adding an override flag or having them to add checksum file.
I checked the history why "--no-check” was added and it was part of bf902a810f98f55dd9e8cb9e6c6b0903f9902157
What's your preference?
Based on review verdict on oe-core patch, I will send patch v2 here.
Regards,
Jaipaul Cheernam
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Tuesday, 16 June 2026 at 12:07
To: Jaipaul Cheernam <jaipaul.cheernam@est.tech>; bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Subject: Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
On Mon, 2026-06-15 at 14:18 +0200, Jaipaul Cheernam via lists.openembedded.org wrote:
> Allow distro-specific buildtools to be configured with a custom URL,
> filename, and SDK name via an install-buildtools section in
> config-upstream.json. Expose the same options as CLI arguments: --url,
> --filename, --sdk-name, --no-check. CLI arguments take precedence over
> config values.
>
> This requires the --sdk-name option to be available in
> oe-scripts/install-buildtools from openembedded-core.
I replied to the oe-core patch, lets just remove the need to have --
sdk-name, I don't think we need that. I also don't understand why you
want/need the --no-check option, we should be able to tell if we should
be checking the checksum or not? If that means fixing the install
script, so be it.
Cheers,
Richard
[-- Attachment #2: Type: text/html, Size: 5449 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-16 14:18 ` Jaipaul Cheernam
@ 2026-06-16 14:26 ` Alexander Kanavin
2026-06-17 10:30 ` Jaipaul Cheernam
0 siblings, 1 reply; 19+ messages in thread
From: Alexander Kanavin @ 2026-06-16 14:26 UTC (permalink / raw)
To: jaipaul.cheernam; +Cc: Richard Purdie, bitbake-devel@lists.openembedded.org
On Tue, 16 Jun 2026 at 16:18, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Agreed on removing --sdk-name and I have sent Patch v2 for review in oe-core.
>
> For --no-check — I'm fine removing it and having the install-buildtools auto-detect, but we need to decide on the failure mode: if the checksum file can't be fetched
> (404 or network error), should we:
>
> (a) warn and continue the install without verification, or
> (b) hard fail and refuse to install?
>
> If (a), we're effectively auto---no-check on failure, which may silently skip validation on transient network issues. If (b), users with custom/local
> URLs that don't host checksum files would have no way to install without us re-adding an override flag or having them to add checksum file.
>
> I checked the history why "--no-check” was added and it was part of bf902a810f98f55dd9e8cb9e6c6b0903f9902157
>
> What's your preference?
Please note my comment in this thread. If we use bitbake fetcher with
checksum in a bitbake-setup config file, then the checksum would be
enforced in a way that is secure against tampering or accidental
replacement.
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-16 14:26 ` Alexander Kanavin
@ 2026-06-17 10:30 ` Jaipaul Cheernam
2026-06-17 13:28 ` Alexander Kanavin
0 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-17 10:30 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: Richard Purdie, bitbake-devel@lists.openembedded.org
[-- Attachment #1: Type: text/plain, Size: 4422 bytes --]
Hi Alex and Richard,
I am in favour of this approach. Here's what I'm thinking for the implementation:
1. bitbake-setup downloads the buildtools installer using bb.fetch, with sha256sum specified in the bitbake-setup config (same mechanism as layers). This
gives us download caching in DL_DIR and checksum enforcement for free.
2. bitbake-setup then calls install-buildtools --url file://<DL_DIR> --filename <name> --no-check — the --no-check is safe because bb.fetch already
validated the sha256. (By writing sha256sum from config into DL_DIR , we can avoid --no-check as well but bit a hacky )
Two options for install-buildtools here:
(a) Update install-buildtools to handle file:// URLs natively — uses the file in-place, avoids a redundant copy of a potentially large (~500MB)
installer to a temp directory.
(b) Leave install-buildtools unchanged — wget already supports file:// and will copy the file to temp. t requires zero changes to
install-buildtools, making this a bitbake-only patch.
I'm fine with either. What's your preference?
Pros:
- Buildtools installer is cached in DL_DIR alongside layers and source tarballs — survives upstream disappearing
- Checksum is in a trusted config file, not fetched from the same server as the installer — protects against supply chain attacks
- Consistent with how everything else is fetched in bitbake-setup
- --no-check remains in install-buildtools but is justified — the caller (bitbake-setup) has already done validation via bb.fetch
Concerns:
- If sha256sum is omitted from config, bb.fetch still downloads but prints a warning about missing checksum (respects BB_STRICT_CHECKSUM if set)
Flow when user runs bitbake-setup install-buildtools --setup-dir ./my-setup:
1. bitbake-setup reads config → gets url, filename, sha256sum
2. Constructs SRC_URI: "https://downloads.../x86_64-buildtools-...5.3.2.sh;sha256sum=a1b2c3..."
3. bb.fetch.Fetch downloads to DL_DIR, validates sha256 (hard fail if mismatch)
4. Calls: install-buildtools --url file://<DL_DIR> --filename <name> --no-check -d ./my-setup/buildtools
5. install-buildtools uses file, makes executable, runs it, sets up env
Config format:
"install-buildtools": {
"url": "https://downloads.yoctoproject.org/releases/yocto/yocto-5.3.2/buildtools",
"filename": "x86_64-buildtools-extended-nativesdk-standalone-5.3.2.sh",
"sha256sum": "a1b2c3d4..."
}
If sha256sum is omitted — bb.fetch downloads with a warning (user's choice).
If sha256sum doesn't match — bb.fetch raises FetchError before install-buildtools is ever called.
Standalone usage of install-buildtools remains unchanged.
Does this match what you had in mind?
Regards,
Jaipaul Cheernam
From: Alexander Kanavin <alex.kanavin@gmail.com>
Date: Tuesday, 16 June 2026 at 16:26
To: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Subject: Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
On Tue, 16 Jun 2026 at 16:18, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Agreed on removing --sdk-name and I have sent Patch v2 for review in oe-core.
>
> For --no-check — I'm fine removing it and having the install-buildtools auto-detect, but we need to decide on the failure mode: if the checksum file can't be fetched
> (404 or network error), should we:
>
> (a) warn and continue the install without verification, or
> (b) hard fail and refuse to install?
>
> If (a), we're effectively auto---no-check on failure, which may silently skip validation on transient network issues. If (b), users with custom/local
> URLs that don't host checksum files would have no way to install without us re-adding an override flag or having them to add checksum file.
>
> I checked the history why "--no-check” was added and it was part of bf902a810f98f55dd9e8cb9e6c6b0903f9902157
>
> What's your preference?
Please note my comment in this thread. If we use bitbake fetcher with
checksum in a bitbake-setup config file, then the checksum would be
enforced in a way that is secure against tampering or accidental
replacement.
Alex
[-- Attachment #2: Type: text/html, Size: 12310 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH] bitbake-setup: pass install-buildtools options from config and CLI
2026-06-17 10:30 ` Jaipaul Cheernam
@ 2026-06-17 13:28 ` Alexander Kanavin
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Kanavin @ 2026-06-17 13:28 UTC (permalink / raw)
To: Jaipaul Cheernam; +Cc: Richard Purdie, bitbake-devel@lists.openembedded.org
On Wed, 17 Jun 2026 at 12:30, Jaipaul Cheernam
<jaipaul.cheernam@est.tech> wrote:
> Two options for install-buildtools here:
>
> (a) Update install-buildtools to handle file:// URLs natively — uses the file in-place, avoids a redundant copy of a potentially large (~500MB)
> installer to a temp directory.
>
> (b) Leave install-buildtools unchanged — wget already supports file:// and will copy the file to temp. t requires zero changes to
> install-buildtools, making this a bitbake-only patch.
>
> I'm fine with either. What's your preference?
I like option (a). install-buildtools is in oe-core, and can be
changed if there's a use case. It can be modified to take a new option
--local-file that simply takes a file path and doesn't perform
checksum verification at all.
> Config format:
>
> "install-buildtools": {
> "url": "https://downloads.yoctoproject.org/releases/yocto/yocto-5.3.2/buildtools",
> "filename": "x86_64-buildtools-extended-nativesdk-standalone-5.3.2.sh",
> "sha256sum": "a1b2c3d4..."
> }
url and filename could be combined into one complete url field, no
need to split them, as they're not anymore given to install-buildtools
script.
> If sha256sum is omitted — bb.fetch downloads with a warning (user's choice).
I'd say absence of sha256sum should be a hard error, with a hint about
how to add the sum to the file.
> If sha256sum doesn't match — bb.fetch raises FetchError before install-buildtools is ever called.
>
> Standalone usage of install-buildtools remains unchanged.
>
> Does this match what you had in mind?
Yes, more or less :-)
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-15 12:18 [PATCH] bitbake-setup: pass install-buildtools options from config and CLI Jaipaul Cheernam
2026-06-15 13:31 ` [bitbake-devel] " Alexander Kanavin
2026-06-16 10:06 ` Richard Purdie
@ 2026-06-26 13:43 ` Jaipaul Cheernam
2026-06-26 16:48 ` [bitbake-devel] " Alexander Kanavin
2026-06-26 19:23 ` [PATCH v3] " Jaipaul Cheernam
` (2 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-26 13:43 UTC (permalink / raw)
To: bitbake-devel; +Cc: Jaipaul Cheernam
When url and sha256sum are configured, download the buildtools
installer via bb.fetch with checksum enforcement. The file is
cached in DL_DIR and passed to install-buildtools via --local-file.
If no install-buildtools config is present, fall back to calling
install-buildtools directly (preserving existing default behaviour).
CLI overrides available via --url and --sha256.
Depends: [oe-core] install-buildtools: add --local-file option
AI-Generated: Kiro with Claude Opus 4.6
Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
---
Changes since v1:
- Use bb.fetch with mandatory sha256sum when configured
- Fall back to install-buildtools defaults when no config present
- Explicit sha256 verification after download (covers file:// and all fetchers)
- Combined url+filename into single url field
- Removed --sdk-name, --filename, --no-check/--check
- Added --url and --sha256 CLI overrides
- Updated schema, docs, and tests
bin/bitbake-setup | 91 ++++++++++++++++++-
.../bitbake-user-manual-environment-setup.rst | 19 ++++
lib/bb/tests/setup.py | 85 ++++++++++++++++-
setup-schema/bitbake-setup.schema.json | 21 +++++
4 files changed, 208 insertions(+), 8 deletions(-)
diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 97ea08d11..461932b26 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -1053,10 +1053,91 @@ def install_buildtools(top_dir, settings, args, d):
return
shutil.rmtree(buildtools_install_dir)
- install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
- buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
- logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
- subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
+ install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+
+ buildtools_config = {}
+ upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
+ try:
+ with open(upstream_config_path) as f:
+ upstream_config = json.load(f)
+ buildtools_config = upstream_config.get('data', {}).get(
+ 'bitbake-setup', {}).get('install-buildtools', {})
+ except FileNotFoundError:
+ pass
+ except json.JSONDecodeError as e:
+ logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
+ sys.exit(1)
+
+ url = args.url if args.url is not None else buildtools_config.get('url')
+ sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum')
+
+ if not url and not sha256:
+ # No config provided -- fall back to install-buildtools defaults
+ logger.plain("No install-buildtools config found, using script defaults")
+ buildtools_download_dir = add_unique_timestamp_to_path(
+ os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+ subprocess.check_call([install_buildtools_script,
+ '-d', buildtools_install_dir,
+ '--downloads-directory', buildtools_download_dir])
+ return
+
+ if not url:
+ logger.error("No buildtools URL configured. "
+ "Set 'url' in the install-buildtools config section or pass --url.")
+ sys.exit(1)
+
+ if not sha256:
+ logger.error("No sha256sum configured for buildtools installer. "
+ "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n"
+ "You can obtain it with: sha256sum <installer-file>")
+ sys.exit(1)
+
+ # Download using bb.fetch with sha256 validation
+ src_uri = "{};sha256sum={}".format(url, sha256)
+
+ logger.plain("Fetching buildtools installer using bitbake fetcher")
+ logger.plain(" {}".format(url))
+
+ fetcher = bb.fetch.Fetch([src_uri], d)
+ try:
+ fetcher.download()
+ except bb.fetch2.ChecksumError as e:
+ logger.error("Checksum mismatch for buildtools installer. "
+ "Verify the URL and downloaded file.\n%s" % str(e))
+ sys.exit(1)
+ except bb.fetch2.FetchError as e:
+ logger.error("Failed to download buildtools installer: %s" % str(e))
+ sys.exit(1)
+
+ # Locate the downloaded file
+ local_path = fetcher.localpath(src_uri)
+
+ if not os.path.exists(local_path):
+ logger.error("Downloaded file not found at expected location")
+ sys.exit(1)
+
+ # Verify checksum explicitly (file:// fetcher does not enforce checksums)
+ actual_sha256 = bb.utils.sha256_file(local_path)
+ if actual_sha256 != sha256:
+ logger.error("Checksum mismatch for buildtools installer:\n"
+ " expected: %s\n"
+ " actual: %s" % (sha256, actual_sha256))
+ sys.exit(1)
+
+ logger.plain("Buildtools installer cached in {} and will be installed into {}".format(
+ os.path.dirname(local_path), buildtools_install_dir))
+
+ cmd = [
+ install_buildtools_script,
+ '--local-file', local_path,
+ '-d', buildtools_install_dir,
+ ]
+
+ try:
+ subprocess.check_call(cmd)
+ except subprocess.CalledProcessError as e:
+ logger.error("install-buildtools failed with exit code %d" % e.returncode)
+ sys.exit(1)
def create_siteconf(top_dir, non_interactive, settings):
siteconfpath = os.path.join(top_dir, 'site.conf')
@@ -1286,6 +1367,8 @@ def main():
parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
add_setup_dir_arg(parser_install_buildtools)
parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
+ parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.')
+ parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.')
parser_install_buildtools.set_defaults(func=install_buildtools)
parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
index aa546c30b..e1bf0404d 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -660,6 +660,25 @@ In addition, the command can take the following arguments:
- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
required if :term:`BBPATH` is already configured.
+- ``--url``: full URL to the buildtools SDK installer. Overrides the value
+ from the configuration file.
+
+- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the
+ value from the configuration file.
+
+When ``url`` and ``sha256sum`` are set in the :term:`Configuration File`,
+the installer is downloaded via ``bb.fetch`` (cached in ``DL_DIR``) and its
+checksum is enforced. If no configuration is present, the script falls back
+to its built-in defaults::
+
+ "bitbake-setup": {
+ "install-buildtools": {
+ "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
+ "sha256sum": "af76648b..."
+ },
+ "configurations": [ ... ]
+ }
+
.. _ref-bbsetup-command-settings:
``bitbake-setup settings``
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index 5592e8196..2c073aefd 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -61,16 +61,31 @@ import getopt
import sys
import os
-opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
+opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "local-file="])
+installdir = None
+local_file = None
for option, value in opts:
if option == '-d':
installdir = value
+ elif option == '--local-file':
+ local_file = value
+ print("install-buildtools local-file={}".format(value))
print("Buildtools installed into {}".format(installdir))
-os.makedirs(installdir)
+os.makedirs(installdir, exist_ok=True)
"""
self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
+ # Dummy buildtools installer for bb.fetch testing
+ self.buildtools_dir = os.path.join(self.tempdir, "buildtools-dl")
+ os.makedirs(self.buildtools_dir)
+ self.buildtools_filename = "x86_64-buildtools-nativesdk-standalone-test.sh"
+ buildtools_filepath = os.path.join(self.buildtools_dir, self.buildtools_filename)
+ with open(buildtools_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho dummy\n")
+ with open(buildtools_filepath, 'rb') as f:
+ self.buildtools_sha256 = hashlib.sha256(f.read()).hexdigest()
+
bitbakeconfigbuild = """#!/usr/bin/env python3
import os
import sys
@@ -172,11 +187,15 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
"bb-layers-file-relative": ["layerE/meta-layer"],
"oe-fragments": ["test-fragment-2"]
}
- ]
+ ],
+ "install-buildtools": {
+ "url": "file://%s/%s",
+ "sha256sum": "%s"
+ }
},
"version": "1.0"
}
-""" % (sources)
+""" % (sources, self.buildtools_dir, self.buildtools_filename, self.buildtools_sha256)
os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True)
with open(os.path.join(self.registrypath, name), 'w') as f:
f.write(config)
@@ -644,6 +663,64 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
def _count_layer_backups(self, layers_path):
return len([f for f in os.listdir(layers_path) if 'backup' in f])
+ def test_install_buildtools_fetch(self):
+ """Test that install-buildtools uses bb.fetch with sha256 and passes --local-file"""
+ import shutil
+
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ registry_uri = "git://{};protocol=file;branch=master;rev=master".format(
+ self.registrypath)
+ self.runbbsetup(["settings", "set", "default", "registry", registry_uri])
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
+ self.runbbsetup(["init", "--non-interactive", "test-config-bt", "gadget"])
+ setuppath = self.get_setup_path('test-config-bt', 'gadget')
+
+ # test config-driven install (url + sha256sum from config)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
+
+ # test CLI overrides (use a different file to prove precedence)
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ alt_filename = "alt-buildtools-test.sh"
+ alt_filepath = os.path.join(self.buildtools_dir, alt_filename)
+ with open(alt_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho alt\n")
+ with open(alt_filepath, 'rb') as f:
+ alt_sha256 = hashlib.sha256(f.read()).hexdigest()
+ alt_url = "file://{}/{}".format(self.buildtools_dir, alt_filename)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
+ "--url", alt_url,
+ "--sha256", alt_sha256])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertIn(alt_filename, out[0])
+
+ # test wrong sha256 is a hard error
+ with self.assertRaises(bb.process.ExecutionError):
+ self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
+ "--force",
+ "--url", "file://{}/{}".format(
+ self.buildtools_dir, self.buildtools_filename),
+ "--sha256", "bad" * 21 + "b"])
+
+ # test missing sha256sum is a hard error
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'), ignore_errors=True)
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+ config_upstream = json.load(f)
+ config_upstream['data']['bitbake-setup']['install-buildtools'] = {
+ 'url': 'file://{}/{}'.format(self.buildtools_dir, self.buildtools_filename)
+ }
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+ json.dump(config_upstream, f)
+ with self.assertRaises(bb.process.ExecutionError):
+ self.runbbsetup(["install-buildtools", "--setup-dir", setuppath, "--force"])
+
def test_update_rebase_conflicts_strategy(self):
"""Test the --rebase-conflicts-strategy option for the update command.
diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
index 99f47f73d..afd01d5de 100644
--- a/setup-schema/bitbake-setup.schema.json
+++ b/setup-schema/bitbake-setup.schema.json
@@ -141,6 +141,27 @@
},
"additionalProperties": false
}
+ },
+ "install-buildtools": {
+ "type": "object",
+ "description": "Settings for downloading buildtools via bb.fetch",
+ "properties": {
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Full URL to the buildtools SDK installer."
+ },
+ "sha256sum": {
+ "type": "string",
+ "minLength": 1,
+ "description": "SHA256 checksum of the buildtools installer."
+ }
+ },
+ "required": [
+ "url",
+ "sha256sum"
+ ],
+ "additionalProperties": false
}
},
"additionalProperties": false
--
2.34.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-26 13:43 ` [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download Jaipaul Cheernam
@ 2026-06-26 16:48 ` Alexander Kanavin
2026-06-26 19:05 ` Jaipaul Cheernam
0 siblings, 1 reply; 19+ messages in thread
From: Alexander Kanavin @ 2026-06-26 16:48 UTC (permalink / raw)
To: jaipaul.cheernam; +Cc: bitbake-devel
On Fri, 26 Jun 2026 at 15:43, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> When url and sha256sum are configured, download the buildtools
> installer via bb.fetch with checksum enforcement. The file is
> cached in DL_DIR and passed to install-buildtools via --local-file.
Thanks, this basically looks ok, but I have a few review points.
> If no install-buildtools config is present, fall back to calling
> install-buildtools directly (preserving existing default behaviour).
>
> CLI overrides available via --url and --sha256.
I think there should also be a --default command line option, which
would similarly fall back to getting and installing the default
buildtools even if config file specifies something else.
> + # Locate the downloaded file
> + local_path = fetcher.localpath(src_uri)
> +
> + if not os.path.exists(local_path):
> + logger.error("Downloaded file not found at expected location")
> + sys.exit(1)
I would rather use fetcher.unpack(), than use the file directly from
DL_DIR. It's better to only allow the fetcher to touch content of
DL_DIR.
> + # Verify checksum explicitly (file:// fetcher does not enforce checksums)
> + actual_sha256 = bb.utils.sha256_file(local_path)
> + if actual_sha256 != sha256:
> + logger.error("Checksum mismatch for buildtools installer:\n"
> + " expected: %s\n"
> + " actual: %s" % (sha256, actual_sha256))
> + sys.exit(1)
This I don't quite understand. Why is the checksum verified again if
the fetcher already did it? Why is file:// fetcher mentioned? AI
confusion?
> + "bitbake-setup": {
> + "install-buildtools": {
> + "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
> + "sha256sum": "af76648b..."
> + },
> + "configurations": [ ... ]
> + }
I wonder if the 'install-buildtools' block should be defined inside a
particular configuration. Configurations can be nested, so that still
allows defining a single buildtools for all of them, but this would
also allow specifying separate buildtools for particular
configurations. I'm not sure if a use case for that will arise, but at
least bitbake-setup will be ready for it. What you think?
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-26 16:48 ` [bitbake-devel] " Alexander Kanavin
@ 2026-06-26 19:05 ` Jaipaul Cheernam
0 siblings, 0 replies; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-26 19:05 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: bitbake-devel@lists.openembedded.org
[-- Attachment #1: Type: text/plain, Size: 3394 bytes --]
Hi Alex,
Thanks for the review.
1. --default — sure, will add.
2. fetcher. Unpack() — good point, I'll use that instead of reaching into DL_DIR directly
3. The explicit sha256 check — you're right, this was added because automated reviewers pointed out that file://
doesn't enforce checksums. I'll drop the explicit check
4. Putting install-buildtools inside configurations — makes sense for the future. Want me to do that now or as a
follow-up?
I will send v3 with above corrections.
Regards,
Jaipaul Cheernam
🔗 EST Website<https://www.est.tech/>
🔗 EST LinkedIn<https://www.linkedin.com/company/ericsson-software-technology/>
From: Alexander Kanavin <alex.kanavin@gmail.com>
Date: Friday, 26 June 2026 at 18:48
To: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
Cc: bitbake-devel@lists.openembedded.org <bitbake-devel@lists.openembedded.org>
Subject: Re: [bitbake-devel] [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download
On Fri, 26 Jun 2026 at 15:43, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> When url and sha256sum are configured, download the buildtools
> installer via bb.fetch with checksum enforcement. The file is
> cached in DL_DIR and passed to install-buildtools via --local-file.
Thanks, this basically looks ok, but I have a few review points.
> If no install-buildtools config is present, fall back to calling
> install-buildtools directly (preserving existing default behaviour).
>
> CLI overrides available via --url and --sha256.
I think there should also be a --default command line option, which
would similarly fall back to getting and installing the default
buildtools even if config file specifies something else.
> + # Locate the downloaded file
> + local_path = fetcher.localpath(src_uri)
> +
> + if not os.path.exists(local_path):
> + logger.error("Downloaded file not found at expected location")
> + sys.exit(1)
I would rather use fetcher.unpack(), than use the file directly from
DL_DIR. It's better to only allow the fetcher to touch content of
DL_DIR.
> + # Verify checksum explicitly (file:// fetcher does not enforce checksums)
> + actual_sha256 = bb.utils.sha256_file(local_path)
> + if actual_sha256 != sha256:
> + logger.error("Checksum mismatch for buildtools installer:\n"
> + " expected: %s\n"
> + " actual: %s" % (sha256, actual_sha256))
> + sys.exit(1)
This I don't quite understand. Why is the checksum verified again if
the fetcher already did it? Why is file:// fetcher mentioned? AI
confusion?
> + "bitbake-setup": {
> + "install-buildtools": {
> + "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
> + "sha256sum": "af76648b..."
> + },
> + "configurations": [ ... ]
> + }
I wonder if the 'install-buildtools' block should be defined inside a
particular configuration. Configurations can be nested, so that still
allows defining a single buildtools for all of them, but this would
also allow specifying separate buildtools for particular
configurations. I'm not sure if a use case for that will arise, but at
least bitbake-setup will be ready for it. What you think?
Alex
[-- Attachment #2: Type: text/html, Size: 8506 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v3] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-15 12:18 [PATCH] bitbake-setup: pass install-buildtools options from config and CLI Jaipaul Cheernam
` (2 preceding siblings ...)
2026-06-26 13:43 ` [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download Jaipaul Cheernam
@ 2026-06-26 19:23 ` Jaipaul Cheernam
2026-06-26 19:44 ` [bitbake-devel] " Alexander Kanavin
2026-06-29 20:34 ` [PATCH v4] " Jaipaul Cheernam
2026-06-30 12:02 ` [PATCH v5] " Jaipaul Cheernam
5 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-26 19:23 UTC (permalink / raw)
To: bitbake-devel; +Cc: Jaipaul Cheernam
When url and sha256sum are configured, download the buildtools
installer via bb.fetch with checksum enforcement. The file is
unpacked to a working directory and passed to install-buildtools
via --local-file.
If no config is present or --default is passed, fall back to
calling install-buildtools directly with its built-in defaults.
CLI overrides available via --url and --sha256.
Depends: [oe-core] install-buildtools: add --local-file option
AI-Generated: Kiro with Claude Opus 4.6
Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
---
Changes since v2:
- Added --default to bypass config and use script defaults
- Use fetcher.unpack() instead of reading from DL_DIR directly
- Removed explicit sha256 verification (fetcher handles it for https)
- Per-configuration install-buildtools config takes precedence over top-level
Changes since v1:
- Use bb.fetch with mandatory sha256sum when configured
- Fall back to install-buildtools defaults when no config present
- Combined url+filename into single url field
- Removed --sdk-name, --filename, --no-check/--check
- Added --url and --sha256 CLI overrides
- Updated schema, docs, and tests
bin/bitbake-setup | 94 ++++++++++++++++++-
.../bitbake-user-manual-environment-setup.rst | 19 ++++
lib/bb/tests/setup.py | 77 ++++++++++++++-
setup-schema/bitbake-setup.schema.json | 21 +++++
4 files changed, 203 insertions(+), 8 deletions(-)
diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 97ea08d11..aa084538d 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -1053,10 +1053,93 @@ def install_buildtools(top_dir, settings, args, d):
return
shutil.rmtree(buildtools_install_dir)
- install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
- buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
- logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
- subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
+ install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+
+ buildtools_config = {}
+ upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
+ try:
+ with open(upstream_config_path) as f:
+ upstream_config = json.load(f)
+ # Per-configuration takes precedence over top-level
+ buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {})
+ if not buildtools_config:
+ buildtools_config = upstream_config.get('data', {}).get(
+ 'bitbake-setup', {}).get('install-buildtools', {})
+ except FileNotFoundError:
+ pass
+ except json.JSONDecodeError as e:
+ logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
+ sys.exit(1)
+
+ url = args.url if args.url is not None else buildtools_config.get('url')
+ sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum')
+
+ if args.default or (not url and not sha256):
+ # No config or --default: fall back to install-buildtools defaults
+ logger.plain("Using install-buildtools script defaults")
+ buildtools_download_dir = add_unique_timestamp_to_path(
+ os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+ subprocess.check_call([install_buildtools_script,
+ '-d', buildtools_install_dir,
+ '--downloads-directory', buildtools_download_dir])
+ return
+
+ if not url:
+ logger.error("No buildtools URL configured. "
+ "Set 'url' in the install-buildtools config section or pass --url.")
+ sys.exit(1)
+
+ if not sha256:
+ logger.error("No sha256sum configured for buildtools installer. "
+ "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n"
+ "You can obtain it with: sha256sum <installer-file>")
+ sys.exit(1)
+
+ # Download using bb.fetch with sha256 validation
+ src_uri = "{};sha256sum={}".format(url, sha256)
+
+ logger.plain("Fetching buildtools installer using bitbake fetcher")
+ logger.plain(" {}".format(url))
+
+ fetcher = bb.fetch.Fetch([src_uri], d)
+ try:
+ fetcher.download()
+ except bb.fetch2.ChecksumError as e:
+ logger.error("Checksum mismatch for buildtools installer. "
+ "Verify the URL and downloaded file.\n%s" % str(e))
+ sys.exit(1)
+ except bb.fetch2.FetchError as e:
+ logger.error("Failed to download buildtools installer: %s" % str(e))
+ sys.exit(1)
+
+ # Unpack into a working directory (avoid touching DL_DIR directly)
+ import tempfile
+ unpack_base = os.path.join(args.setup_dir, 'buildtools-downloads')
+ os.makedirs(unpack_base, exist_ok=True)
+ unpackdir = tempfile.mkdtemp(dir=unpack_base)
+ fetcher.unpack(unpackdir)
+
+ filename = os.path.basename(url.rstrip('/'))
+ matches = glob.glob(os.path.join(unpackdir, '**', filename), recursive=True)
+ if not matches:
+ logger.error("Unpacked file not found: %s" % filename)
+ sys.exit(1)
+ local_path = matches[0]
+
+ logger.plain("Buildtools installer will be installed into {}".format(
+ buildtools_install_dir))
+
+ cmd = [
+ install_buildtools_script,
+ '--local-file', local_path,
+ '-d', buildtools_install_dir,
+ ]
+
+ try:
+ subprocess.check_call(cmd)
+ except subprocess.CalledProcessError as e:
+ logger.error("install-buildtools failed with exit code %d" % e.returncode)
+ sys.exit(1)
def create_siteconf(top_dir, non_interactive, settings):
siteconfpath = os.path.join(top_dir, 'site.conf')
@@ -1286,6 +1369,9 @@ def main():
parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
add_setup_dir_arg(parser_install_buildtools)
parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
+ parser_install_buildtools.add_argument('--default', action='store_true', help='Use install-buildtools script defaults, ignoring any config.')
+ parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.')
+ parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.')
parser_install_buildtools.set_defaults(func=install_buildtools)
parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
index aa546c30b..e1bf0404d 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -660,6 +660,25 @@ In addition, the command can take the following arguments:
- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
required if :term:`BBPATH` is already configured.
+- ``--url``: full URL to the buildtools SDK installer. Overrides the value
+ from the configuration file.
+
+- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the
+ value from the configuration file.
+
+When ``url`` and ``sha256sum`` are set in the :term:`Configuration File`,
+the installer is downloaded via ``bb.fetch`` (cached in ``DL_DIR``) and its
+checksum is enforced. If no configuration is present, the script falls back
+to its built-in defaults::
+
+ "bitbake-setup": {
+ "install-buildtools": {
+ "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
+ "sha256sum": "af76648b..."
+ },
+ "configurations": [ ... ]
+ }
+
.. _ref-bbsetup-command-settings:
``bitbake-setup settings``
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index 5592e8196..7f07369f6 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -61,16 +61,31 @@ import getopt
import sys
import os
-opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
+opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "local-file="])
+installdir = None
+local_file = None
for option, value in opts:
if option == '-d':
installdir = value
+ elif option == '--local-file':
+ local_file = value
+ print("install-buildtools local-file={}".format(value))
print("Buildtools installed into {}".format(installdir))
-os.makedirs(installdir)
+os.makedirs(installdir, exist_ok=True)
"""
self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
+ # Dummy buildtools installer for bb.fetch testing
+ self.buildtools_dir = os.path.join(self.tempdir, "buildtools-dl")
+ os.makedirs(self.buildtools_dir)
+ self.buildtools_filename = "x86_64-buildtools-nativesdk-standalone-test.sh"
+ buildtools_filepath = os.path.join(self.buildtools_dir, self.buildtools_filename)
+ with open(buildtools_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho dummy\n")
+ with open(buildtools_filepath, 'rb') as f:
+ self.buildtools_sha256 = hashlib.sha256(f.read()).hexdigest()
+
bitbakeconfigbuild = """#!/usr/bin/env python3
import os
import sys
@@ -172,11 +187,15 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
"bb-layers-file-relative": ["layerE/meta-layer"],
"oe-fragments": ["test-fragment-2"]
}
- ]
+ ],
+ "install-buildtools": {
+ "url": "file://%s/%s",
+ "sha256sum": "%s"
+ }
},
"version": "1.0"
}
-""" % (sources)
+""" % (sources, self.buildtools_dir, self.buildtools_filename, self.buildtools_sha256)
os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True)
with open(os.path.join(self.registrypath, name), 'w') as f:
f.write(config)
@@ -644,6 +663,56 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
def _count_layer_backups(self, layers_path):
return len([f for f in os.listdir(layers_path) if 'backup' in f])
+ def test_install_buildtools_fetch(self):
+ """Test that install-buildtools uses bb.fetch with sha256 and passes --local-file"""
+ import shutil
+
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ registry_uri = "git://{};protocol=file;branch=master;rev=master".format(
+ self.registrypath)
+ self.runbbsetup(["settings", "set", "default", "registry", registry_uri])
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
+ self.runbbsetup(["init", "--non-interactive", "test-config-bt", "gadget"])
+ setuppath = self.get_setup_path('test-config-bt', 'gadget')
+
+ # test config-driven install (url + sha256sum from config)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
+
+ # test CLI overrides (use a different file to prove precedence)
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ alt_filename = "alt-buildtools-test.sh"
+ alt_filepath = os.path.join(self.buildtools_dir, alt_filename)
+ with open(alt_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho alt\n")
+ with open(alt_filepath, 'rb') as f:
+ alt_sha256 = hashlib.sha256(f.read()).hexdigest()
+ alt_url = "file://{}/{}".format(self.buildtools_dir, alt_filename)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
+ "--url", alt_url,
+ "--sha256", alt_sha256])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertIn(alt_filename, out[0])
+
+ # test missing sha256sum is a hard error
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'), ignore_errors=True)
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+ config_upstream = json.load(f)
+ config_upstream['data']['bitbake-setup']['install-buildtools'] = {
+ 'url': 'file://{}/{}'.format(self.buildtools_dir, self.buildtools_filename)
+ }
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+ json.dump(config_upstream, f)
+ with self.assertRaises(bb.process.ExecutionError):
+ self.runbbsetup(["install-buildtools", "--setup-dir", setuppath, "--force"])
+
def test_update_rebase_conflicts_strategy(self):
"""Test the --rebase-conflicts-strategy option for the update command.
diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
index 99f47f73d..afd01d5de 100644
--- a/setup-schema/bitbake-setup.schema.json
+++ b/setup-schema/bitbake-setup.schema.json
@@ -141,6 +141,27 @@
},
"additionalProperties": false
}
+ },
+ "install-buildtools": {
+ "type": "object",
+ "description": "Settings for downloading buildtools via bb.fetch",
+ "properties": {
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Full URL to the buildtools SDK installer."
+ },
+ "sha256sum": {
+ "type": "string",
+ "minLength": 1,
+ "description": "SHA256 checksum of the buildtools installer."
+ }
+ },
+ "required": [
+ "url",
+ "sha256sum"
+ ],
+ "additionalProperties": false
}
},
"additionalProperties": false
--
2.34.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH v3] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-26 19:23 ` [PATCH v3] " Jaipaul Cheernam
@ 2026-06-26 19:44 ` Alexander Kanavin
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Kanavin @ 2026-06-26 19:44 UTC (permalink / raw)
To: jaipaul.cheernam; +Cc: bitbake-devel
On Fri, 26 Jun 2026 at 21:23, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> + # Per-configuration takes precedence over top-level
> + buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {})
> + if not buildtools_config:
> + buildtools_config = upstream_config.get('data', {}).get(
> + 'bitbake-setup', {}).get('install-buildtools', {})
This wasn't quite the idea. I wanted to make per-configuration the
only place to specify custom buildtools, so no top level support at
all.
> + # Unpack into a working directory (avoid touching DL_DIR directly)
> + import tempfile
> + unpack_base = os.path.join(args.setup_dir, 'buildtools-downloads')
> + os.makedirs(unpack_base, exist_ok=True)
> + unpackdir = tempfile.mkdtemp(dir=unpack_base)
> + fetcher.unpack(unpackdir)
This should be using the same location as default buildtools, e.g.
fecher.unpack(buildtools_download_dir). So it's consistent, if someone
is expecting to find the buildtools archive there.
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v4] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-15 12:18 [PATCH] bitbake-setup: pass install-buildtools options from config and CLI Jaipaul Cheernam
` (3 preceding siblings ...)
2026-06-26 19:23 ` [PATCH v3] " Jaipaul Cheernam
@ 2026-06-29 20:34 ` Jaipaul Cheernam
2026-06-30 11:47 ` [bitbake-devel] " Alexander Kanavin
2026-06-30 12:02 ` [PATCH v5] " Jaipaul Cheernam
5 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-29 20:34 UTC (permalink / raw)
To: bitbake-devel; +Cc: Jaipaul Cheernam
When url and sha256sum are configured in the selected configuration,
download the buildtools installer via bb.fetch with checksum
enforcement. The file is unpacked to the standard buildtools download
location and passed to install-buildtools via --local-file.
If no config is present or --default is passed, fall back to
calling install-buildtools directly with its built-in defaults.
CLI overrides available via --url and --sha256.
Depends: [oe-core] install-buildtools: add --local-file option
AI-Generated: Kiro with Claude Opus 4.6
Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
---
Changes since v3:
- install-buildtools config is per-configuration only (no top-level)
- Use buildtools_download_dir for unpack (consistent with default path)
- Updated schema, docs, and tests to match
Changes since v2:
- Added --default to bypass config and use script defaults
- Use fetcher.unpack() instead of reading from DL_DIR directly
- Removed explicit sha256 verification (fetcher handles it for https)
Changes since v1:
- Use bb.fetch with mandatory sha256sum when configured
- Fall back to install-buildtools defaults when no config present
- Combined url+filename into single url field
- Removed --sdk-name, --filename, --no-check/--check
- Added --url and --sha256 CLI overrides
- Updated schema, docs, and tests
bin/bitbake-setup | 89 ++++++++++++++++++-
.../bitbake-user-manual-environment-setup.rst | 24 +++++
lib/bb/tests/setup.py | 77 +++++++++++++++-
setup-schema/bitbake-setup.schema.json | 21 +++++
4 files changed, 203 insertions(+), 8 deletions(-)
diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 97ea08d11..9a3507c1e 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -1053,10 +1053,88 @@ def install_buildtools(top_dir, settings, args, d):
return
shutil.rmtree(buildtools_install_dir)
- install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
- buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
- logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
- subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
+ install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+
+ buildtools_config = {}
+ upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
+ try:
+ with open(upstream_config_path) as f:
+ upstream_config = json.load(f)
+ buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {})
+ except FileNotFoundError:
+ pass
+ except json.JSONDecodeError as e:
+ logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
+ sys.exit(1)
+
+ url = args.url if args.url is not None else buildtools_config.get('url')
+ sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum')
+
+ if args.default or (not url and not sha256):
+ # No config or --default: fall back to install-buildtools defaults
+ logger.plain("Using install-buildtools script defaults")
+ buildtools_download_dir = add_unique_timestamp_to_path(
+ os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+ subprocess.check_call([install_buildtools_script,
+ '-d', buildtools_install_dir,
+ '--downloads-directory', buildtools_download_dir])
+ return
+
+ if not url:
+ logger.error("No buildtools URL configured. "
+ "Set 'url' in the install-buildtools config section or pass --url.")
+ sys.exit(1)
+
+ if not sha256:
+ logger.error("No sha256sum configured for buildtools installer. "
+ "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n"
+ "You can obtain it with: sha256sum <installer-file>")
+ sys.exit(1)
+
+ # Download using bb.fetch with sha256 validation
+ src_uri = "{};sha256sum={}".format(url, sha256)
+
+ logger.plain("Fetching buildtools installer using bitbake fetcher")
+ logger.plain(" {}".format(url))
+
+ fetcher = bb.fetch.Fetch([src_uri], d)
+ try:
+ fetcher.download()
+ except bb.fetch2.ChecksumError as e:
+ logger.error("Checksum mismatch for buildtools installer. "
+ "Verify the URL and downloaded file.\n%s" % str(e))
+ sys.exit(1)
+ except bb.fetch2.FetchError as e:
+ logger.error("Failed to download buildtools installer: %s" % str(e))
+ sys.exit(1)
+
+ # Unpack into the standard buildtools download location
+ buildtools_download_dir = add_unique_timestamp_to_path(
+ os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+ os.makedirs(buildtools_download_dir, exist_ok=True)
+ fetcher.unpack(buildtools_download_dir)
+
+ filename = os.path.basename(url.rstrip('/'))
+ matches = glob.glob(os.path.join(buildtools_download_dir, '**', filename), recursive=True)
+ if not matches:
+ logger.error("Unpacked file not found: %s" % filename)
+ sys.exit(1)
+ local_path = matches[0]
+
+ logger.plain("Buildtools installer will be installed into {}".format(
+ buildtools_install_dir))
+
+ cmd = [
+ install_buildtools_script,
+ '--local-file', local_path,
+ '-d', buildtools_install_dir,
+ ]
+
+ try:
+ subprocess.check_call(cmd)
+ except subprocess.CalledProcessError as e:
+ logger.error("install-buildtools failed with exit code %d" % e.returncode)
+ sys.exit(1)
def create_siteconf(top_dir, non_interactive, settings):
siteconfpath = os.path.join(top_dir, 'site.conf')
@@ -1286,6 +1364,9 @@ def main():
parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
add_setup_dir_arg(parser_install_buildtools)
parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
+ parser_install_buildtools.add_argument('--default', action='store_true', help='Use install-buildtools script defaults, ignoring any config.')
+ parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.')
+ parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.')
parser_install_buildtools.set_defaults(func=install_buildtools)
parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
index aa546c30b..e1701555e 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -660,6 +660,30 @@ In addition, the command can take the following arguments:
- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
required if :term:`BBPATH` is already configured.
+- ``--url``: full URL to the buildtools SDK installer. Overrides the value
+ from the configuration file.
+
+- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the
+ value from the configuration file.
+
+When ``url`` and ``sha256sum`` are set in the :term:`Configuration File`,
+the installer is downloaded via ``bb.fetch`` (cached in ``DL_DIR``) and its
+checksum is enforced. If no configuration is present, the script falls back
+to its built-in defaults::
+
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "my-config",
+ "install-buildtools": {
+ "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
+ "sha256sum": "af76648b..."
+ },
+ ...
+ }
+ ]
+ }
+
.. _ref-bbsetup-command-settings:
``bitbake-setup settings``
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index 5592e8196..8ca64cb50 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -61,16 +61,31 @@ import getopt
import sys
import os
-opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
+opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "local-file="])
+installdir = None
+local_file = None
for option, value in opts:
if option == '-d':
installdir = value
+ elif option == '--local-file':
+ local_file = value
+ print("install-buildtools local-file={}".format(value))
print("Buildtools installed into {}".format(installdir))
-os.makedirs(installdir)
+os.makedirs(installdir, exist_ok=True)
"""
self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
+ # Dummy buildtools installer for bb.fetch testing
+ self.buildtools_dir = os.path.join(self.tempdir, "buildtools-dl")
+ os.makedirs(self.buildtools_dir)
+ self.buildtools_filename = "x86_64-buildtools-nativesdk-standalone-test.sh"
+ buildtools_filepath = os.path.join(self.buildtools_dir, self.buildtools_filename)
+ with open(buildtools_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho dummy\n")
+ with open(buildtools_filepath, 'rb') as f:
+ self.buildtools_sha256 = hashlib.sha256(f.read()).hexdigest()
+
bitbakeconfigbuild = """#!/usr/bin/env python3
import os
import sys
@@ -108,7 +123,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
"name": "gadget",
"description": "Gadget configuration",
"oe-template": "test-configuration-gadget",
- "oe-fragments": ["test-fragment-1"]
+ "oe-fragments": ["test-fragment-1"],
+ "install-buildtools": {
+ "url": "file://%s/%s",
+ "sha256sum": "%s"
+ }
},
{
"name": "gizmo",
@@ -176,7 +195,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
},
"version": "1.0"
}
-""" % (sources)
+""" % (sources, self.buildtools_dir, self.buildtools_filename, self.buildtools_sha256)
os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True)
with open(os.path.join(self.registrypath, name), 'w') as f:
f.write(config)
@@ -644,6 +663,56 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
def _count_layer_backups(self, layers_path):
return len([f for f in os.listdir(layers_path) if 'backup' in f])
+ def test_install_buildtools_fetch(self):
+ """Test that install-buildtools uses bb.fetch with sha256 and passes --local-file"""
+ import shutil
+
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ registry_uri = "git://{};protocol=file;branch=master;rev=master".format(
+ self.registrypath)
+ self.runbbsetup(["settings", "set", "default", "registry", registry_uri])
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
+ self.runbbsetup(["init", "--non-interactive", "test-config-bt", "gadget"])
+ setuppath = self.get_setup_path('test-config-bt', 'gadget')
+
+ # test config-driven install (url + sha256sum from config)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
+
+ # test CLI overrides (use a different file to prove precedence)
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ alt_filename = "alt-buildtools-test.sh"
+ alt_filepath = os.path.join(self.buildtools_dir, alt_filename)
+ with open(alt_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho alt\n")
+ with open(alt_filepath, 'rb') as f:
+ alt_sha256 = hashlib.sha256(f.read()).hexdigest()
+ alt_url = "file://{}/{}".format(self.buildtools_dir, alt_filename)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
+ "--url", alt_url,
+ "--sha256", alt_sha256])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertIn(alt_filename, out[0])
+
+ # test missing sha256sum is a hard error
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'), ignore_errors=True)
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+ config_upstream = json.load(f)
+ config_upstream['bitbake-config']['install-buildtools'] = {
+ 'url': 'file://{}/{}'.format(self.buildtools_dir, self.buildtools_filename)
+ }
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+ json.dump(config_upstream, f)
+ with self.assertRaises(bb.process.ExecutionError):
+ self.runbbsetup(["install-buildtools", "--setup-dir", setuppath, "--force"])
+
def test_update_rebase_conflicts_strategy(self):
"""Test the --rebase-conflicts-strategy option for the update command.
diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
index 99f47f73d..797d45252 100644
--- a/setup-schema/bitbake-setup.schema.json
+++ b/setup-schema/bitbake-setup.schema.json
@@ -137,6 +137,27 @@
"setup-dir-name": {
"type": "string",
"description": "A suggestion for the setup directory name, $-prefixed keys from oe-fragments-one-of will be substituted with user selections."
+ },
+ "install-buildtools": {
+ "type": "object",
+ "description": "Settings for downloading buildtools via bb.fetch",
+ "properties": {
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Full URL to the buildtools SDK installer."
+ },
+ "sha256sum": {
+ "type": "string",
+ "minLength": 1,
+ "description": "SHA256 checksum of the buildtools installer."
+ }
+ },
+ "required": [
+ "url",
+ "sha256sum"
+ ],
+ "additionalProperties": false
}
},
"additionalProperties": false
--
2.34.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH v4] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-29 20:34 ` [PATCH v4] " Jaipaul Cheernam
@ 2026-06-30 11:47 ` Alexander Kanavin
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Kanavin @ 2026-06-30 11:47 UTC (permalink / raw)
To: jaipaul.cheernam; +Cc: bitbake-devel
Thanks, I think this is good overall. Perhaps there's no need to set
buildtools_download_dir twice in the same way in two different code
paths, but that's a minor issue. The patch should go to general review
and CI now.
Alex
On Mon, 29 Jun 2026 at 22:34, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
>
> When url and sha256sum are configured in the selected configuration,
> download the buildtools installer via bb.fetch with checksum
> enforcement. The file is unpacked to the standard buildtools download
> location and passed to install-buildtools via --local-file.
>
> If no config is present or --default is passed, fall back to
> calling install-buildtools directly with its built-in defaults.
>
> CLI overrides available via --url and --sha256.
>
> Depends: [oe-core] install-buildtools: add --local-file option
> AI-Generated: Kiro with Claude Opus 4.6
> Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
> ---
>
> Changes since v3:
> - install-buildtools config is per-configuration only (no top-level)
> - Use buildtools_download_dir for unpack (consistent with default path)
> - Updated schema, docs, and tests to match
>
> Changes since v2:
> - Added --default to bypass config and use script defaults
> - Use fetcher.unpack() instead of reading from DL_DIR directly
> - Removed explicit sha256 verification (fetcher handles it for https)
>
> Changes since v1:
> - Use bb.fetch with mandatory sha256sum when configured
> - Fall back to install-buildtools defaults when no config present
> - Combined url+filename into single url field
> - Removed --sdk-name, --filename, --no-check/--check
> - Added --url and --sha256 CLI overrides
> - Updated schema, docs, and tests
> bin/bitbake-setup | 89 ++++++++++++++++++-
> .../bitbake-user-manual-environment-setup.rst | 24 +++++
> lib/bb/tests/setup.py | 77 +++++++++++++++-
> setup-schema/bitbake-setup.schema.json | 21 +++++
> 4 files changed, 203 insertions(+), 8 deletions(-)
>
> diff --git a/bin/bitbake-setup b/bin/bitbake-setup
> index 97ea08d11..9a3507c1e 100755
> --- a/bin/bitbake-setup
> +++ b/bin/bitbake-setup
> @@ -1053,10 +1053,88 @@ def install_buildtools(top_dir, settings, args, d):
> return
> shutil.rmtree(buildtools_install_dir)
>
> - install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
> - buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
> - logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
> - subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
> + install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
> +
> + buildtools_config = {}
> + upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
> + try:
> + with open(upstream_config_path) as f:
> + upstream_config = json.load(f)
> + buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {})
> + except FileNotFoundError:
> + pass
> + except json.JSONDecodeError as e:
> + logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
> + sys.exit(1)
> +
> + url = args.url if args.url is not None else buildtools_config.get('url')
> + sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum')
> +
> + if args.default or (not url and not sha256):
> + # No config or --default: fall back to install-buildtools defaults
> + logger.plain("Using install-buildtools script defaults")
> + buildtools_download_dir = add_unique_timestamp_to_path(
> + os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
> + subprocess.check_call([install_buildtools_script,
> + '-d', buildtools_install_dir,
> + '--downloads-directory', buildtools_download_dir])
> + return
> +
> + if not url:
> + logger.error("No buildtools URL configured. "
> + "Set 'url' in the install-buildtools config section or pass --url.")
> + sys.exit(1)
> +
> + if not sha256:
> + logger.error("No sha256sum configured for buildtools installer. "
> + "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n"
> + "You can obtain it with: sha256sum <installer-file>")
> + sys.exit(1)
> +
> + # Download using bb.fetch with sha256 validation
> + src_uri = "{};sha256sum={}".format(url, sha256)
> +
> + logger.plain("Fetching buildtools installer using bitbake fetcher")
> + logger.plain(" {}".format(url))
> +
> + fetcher = bb.fetch.Fetch([src_uri], d)
> + try:
> + fetcher.download()
> + except bb.fetch2.ChecksumError as e:
> + logger.error("Checksum mismatch for buildtools installer. "
> + "Verify the URL and downloaded file.\n%s" % str(e))
> + sys.exit(1)
> + except bb.fetch2.FetchError as e:
> + logger.error("Failed to download buildtools installer: %s" % str(e))
> + sys.exit(1)
> +
> + # Unpack into the standard buildtools download location
> + buildtools_download_dir = add_unique_timestamp_to_path(
> + os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
> + os.makedirs(buildtools_download_dir, exist_ok=True)
> + fetcher.unpack(buildtools_download_dir)
> +
> + filename = os.path.basename(url.rstrip('/'))
> + matches = glob.glob(os.path.join(buildtools_download_dir, '**', filename), recursive=True)
> + if not matches:
> + logger.error("Unpacked file not found: %s" % filename)
> + sys.exit(1)
> + local_path = matches[0]
> +
> + logger.plain("Buildtools installer will be installed into {}".format(
> + buildtools_install_dir))
> +
> + cmd = [
> + install_buildtools_script,
> + '--local-file', local_path,
> + '-d', buildtools_install_dir,
> + ]
> +
> + try:
> + subprocess.check_call(cmd)
> + except subprocess.CalledProcessError as e:
> + logger.error("install-buildtools failed with exit code %d" % e.returncode)
> + sys.exit(1)
>
> def create_siteconf(top_dir, non_interactive, settings):
> siteconfpath = os.path.join(top_dir, 'site.conf')
> @@ -1286,6 +1364,9 @@ def main():
> parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
> add_setup_dir_arg(parser_install_buildtools)
> parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
> + parser_install_buildtools.add_argument('--default', action='store_true', help='Use install-buildtools script defaults, ignoring any config.')
> + parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.')
> + parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.')
> parser_install_buildtools.set_defaults(func=install_buildtools)
>
> parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> index aa546c30b..e1701555e 100644
> --- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> +++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> @@ -660,6 +660,30 @@ In addition, the command can take the following arguments:
> - ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
> required if :term:`BBPATH` is already configured.
>
> +- ``--url``: full URL to the buildtools SDK installer. Overrides the value
> + from the configuration file.
> +
> +- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the
> + value from the configuration file.
> +
> +When ``url`` and ``sha256sum`` are set in the :term:`Configuration File`,
> +the installer is downloaded via ``bb.fetch`` (cached in ``DL_DIR``) and its
> +checksum is enforced. If no configuration is present, the script falls back
> +to its built-in defaults::
> +
> + "bitbake-setup": {
> + "configurations": [
> + {
> + "name": "my-config",
> + "install-buildtools": {
> + "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
> + "sha256sum": "af76648b..."
> + },
> + ...
> + }
> + ]
> + }
> +
> .. _ref-bbsetup-command-settings:
>
> ``bitbake-setup settings``
> diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
> index 5592e8196..8ca64cb50 100644
> --- a/lib/bb/tests/setup.py
> +++ b/lib/bb/tests/setup.py
> @@ -61,16 +61,31 @@ import getopt
> import sys
> import os
>
> -opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
> +opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "local-file="])
> +installdir = None
> +local_file = None
> for option, value in opts:
> if option == '-d':
> installdir = value
> + elif option == '--local-file':
> + local_file = value
> + print("install-buildtools local-file={}".format(value))
>
> print("Buildtools installed into {}".format(installdir))
> -os.makedirs(installdir)
> +os.makedirs(installdir, exist_ok=True)
> """
> self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
>
> + # Dummy buildtools installer for bb.fetch testing
> + self.buildtools_dir = os.path.join(self.tempdir, "buildtools-dl")
> + os.makedirs(self.buildtools_dir)
> + self.buildtools_filename = "x86_64-buildtools-nativesdk-standalone-test.sh"
> + buildtools_filepath = os.path.join(self.buildtools_dir, self.buildtools_filename)
> + with open(buildtools_filepath, 'w') as f:
> + f.write("#!/bin/sh\necho dummy\n")
> + with open(buildtools_filepath, 'rb') as f:
> + self.buildtools_sha256 = hashlib.sha256(f.read()).hexdigest()
> +
> bitbakeconfigbuild = """#!/usr/bin/env python3
> import os
> import sys
> @@ -108,7 +123,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
> "name": "gadget",
> "description": "Gadget configuration",
> "oe-template": "test-configuration-gadget",
> - "oe-fragments": ["test-fragment-1"]
> + "oe-fragments": ["test-fragment-1"],
> + "install-buildtools": {
> + "url": "file://%s/%s",
> + "sha256sum": "%s"
> + }
> },
> {
> "name": "gizmo",
> @@ -176,7 +195,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
> },
> "version": "1.0"
> }
> -""" % (sources)
> +""" % (sources, self.buildtools_dir, self.buildtools_filename, self.buildtools_sha256)
> os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True)
> with open(os.path.join(self.registrypath, name), 'w') as f:
> f.write(config)
> @@ -644,6 +663,56 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
> def _count_layer_backups(self, layers_path):
> return len([f for f in os.listdir(layers_path) if 'backup' in f])
>
> + def test_install_buildtools_fetch(self):
> + """Test that install-buildtools uses bb.fetch with sha256 and passes --local-file"""
> + import shutil
> +
> + if 'BBPATH' in os.environ:
> + del os.environ['BBPATH']
> + os.chdir(self.tempdir)
> +
> + registry_uri = "git://{};protocol=file;branch=master;rev=master".format(
> + self.registrypath)
> + self.runbbsetup(["settings", "set", "default", "registry", registry_uri])
> + self.add_file_to_testrepo('test-file', 'initial\n')
> + self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
> + self.runbbsetup(["init", "--non-interactive", "test-config-bt", "gadget"])
> + setuppath = self.get_setup_path('test-config-bt', 'gadget')
> +
> + # test config-driven install (url + sha256sum from config)
> + out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
> + self.assertIn("Buildtools installed into", out[0])
> + self.assertIn("install-buildtools local-file=", out[0])
> + self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
> +
> + # test CLI overrides (use a different file to prove precedence)
> + shutil.rmtree(os.path.join(setuppath, 'buildtools'))
> + alt_filename = "alt-buildtools-test.sh"
> + alt_filepath = os.path.join(self.buildtools_dir, alt_filename)
> + with open(alt_filepath, 'w') as f:
> + f.write("#!/bin/sh\necho alt\n")
> + with open(alt_filepath, 'rb') as f:
> + alt_sha256 = hashlib.sha256(f.read()).hexdigest()
> + alt_url = "file://{}/{}".format(self.buildtools_dir, alt_filename)
> + out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
> + "--url", alt_url,
> + "--sha256", alt_sha256])
> + self.assertIn("Buildtools installed into", out[0])
> + self.assertIn("install-buildtools local-file=", out[0])
> + self.assertIn(alt_filename, out[0])
> +
> + # test missing sha256sum is a hard error
> + shutil.rmtree(os.path.join(setuppath, 'buildtools'), ignore_errors=True)
> + with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
> + config_upstream = json.load(f)
> + config_upstream['bitbake-config']['install-buildtools'] = {
> + 'url': 'file://{}/{}'.format(self.buildtools_dir, self.buildtools_filename)
> + }
> + with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
> + json.dump(config_upstream, f)
> + with self.assertRaises(bb.process.ExecutionError):
> + self.runbbsetup(["install-buildtools", "--setup-dir", setuppath, "--force"])
> +
> def test_update_rebase_conflicts_strategy(self):
> """Test the --rebase-conflicts-strategy option for the update command.
>
> diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
> index 99f47f73d..797d45252 100644
> --- a/setup-schema/bitbake-setup.schema.json
> +++ b/setup-schema/bitbake-setup.schema.json
> @@ -137,6 +137,27 @@
> "setup-dir-name": {
> "type": "string",
> "description": "A suggestion for the setup directory name, $-prefixed keys from oe-fragments-one-of will be substituted with user selections."
> + },
> + "install-buildtools": {
> + "type": "object",
> + "description": "Settings for downloading buildtools via bb.fetch",
> + "properties": {
> + "url": {
> + "type": "string",
> + "minLength": 1,
> + "description": "Full URL to the buildtools SDK installer."
> + },
> + "sha256sum": {
> + "type": "string",
> + "minLength": 1,
> + "description": "SHA256 checksum of the buildtools installer."
> + }
> + },
> + "required": [
> + "url",
> + "sha256sum"
> + ],
> + "additionalProperties": false
> }
> },
> "additionalProperties": false
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#19808): https://lists.openembedded.org/g/bitbake-devel/message/19808
> Mute This Topic: https://lists.openembedded.org/mt/120037327/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v5] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-15 12:18 [PATCH] bitbake-setup: pass install-buildtools options from config and CLI Jaipaul Cheernam
` (4 preceding siblings ...)
2026-06-29 20:34 ` [PATCH v4] " Jaipaul Cheernam
@ 2026-06-30 12:02 ` Jaipaul Cheernam
2026-07-08 15:16 ` Jaipaul Cheernam
5 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-06-30 12:02 UTC (permalink / raw)
To: bitbake-devel; +Cc: Jaipaul Cheernam
When url and sha256sum are configured in the selected configuration,
download the buildtools installer via bb.fetch with checksum
enforcement. The file is unpacked to the standard buildtools download
location and passed to install-buildtools via --local-file.
If no config is present or --default is passed, fall back to
calling install-buildtools directly with its built-in defaults.
CLI overrides available via --url and --sha256.
Depends: [oe-core] install-buildtools: add --local-file option
AI-Generated: Kiro with Claude Opus 4.6
Signed-off-by: Jaipaul Cheernam <jaipaul.cheernam@est.tech>
---
Changes since v4:
- Rebased on latest master
- Deduplicate buildtools_download_dir definition
Changes since v3:
- install-buildtools config is per-configuration only (no top-level)
- Use buildtools_download_dir for unpack (consistent with default path)
- Updated schema, docs, and tests to match
Changes since v2:
- Added --default to bypass config and use script defaults
- Use fetcher.unpack() instead of reading from DL_DIR directly
- Removed explicit sha256 verification (fetcher handles it for https)
Changes since v1:
- Use bb.fetch with mandatory sha256sum when configured
- Fall back to install-buildtools defaults when no config present
- Combined url+filename into single url field
- Removed --sdk-name, --filename, --no-check/--check
- Added --url and --sha256 CLI overrides
- Updated schema, docs, and tests
bin/bitbake-setup | 88 ++++++++++++++++++-
.../bitbake-user-manual-environment-setup.rst | 24 +++++
lib/bb/tests/setup.py | 77 +++++++++++++++-
setup-schema/bitbake-setup.schema.json | 23 ++++-
4 files changed, 203 insertions(+), 9 deletions(-)
diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 97ea08d11..2a0967b19 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -1053,10 +1053,87 @@ def install_buildtools(top_dir, settings, args, d):
return
shutil.rmtree(buildtools_install_dir)
- install_buildtools = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
- buildtools_download_dir = add_unique_timestamp_to_path(os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
- logger.plain("Buildtools archive is downloaded into {} and its content installed into {}".format(buildtools_download_dir, buildtools_install_dir))
- subprocess.check_call("{} -d {} --downloads-directory {}".format(install_buildtools, buildtools_install_dir, buildtools_download_dir), shell=True)
+ install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+
+ buildtools_config = {}
+ upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
+ try:
+ with open(upstream_config_path) as f:
+ upstream_config = json.load(f)
+ buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {})
+ except FileNotFoundError:
+ pass
+ except json.JSONDecodeError as e:
+ logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
+ sys.exit(1)
+
+ url = args.url if args.url is not None else buildtools_config.get('url')
+ sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum')
+
+ buildtools_download_dir = add_unique_timestamp_to_path(
+ os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+
+ if args.default or (not url and not sha256):
+ # No config or --default: fall back to install-buildtools defaults
+ logger.plain("Using install-buildtools script defaults")
+ subprocess.check_call([install_buildtools_script,
+ '-d', buildtools_install_dir,
+ '--downloads-directory', buildtools_download_dir])
+ return
+
+ if not url:
+ logger.error("No buildtools URL configured. "
+ "Set 'url' in the install-buildtools config section or pass --url.")
+ sys.exit(1)
+
+ if not sha256:
+ logger.error("No sha256sum configured for buildtools installer. "
+ "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n"
+ "You can obtain it with: sha256sum <installer-file>")
+ sys.exit(1)
+
+ # Download using bb.fetch with sha256 validation
+ src_uri = "{};sha256sum={}".format(url, sha256)
+
+ logger.plain("Fetching buildtools installer using bitbake fetcher")
+ logger.plain(" {}".format(url))
+
+ fetcher = bb.fetch.Fetch([src_uri], d)
+ try:
+ fetcher.download()
+ except bb.fetch2.ChecksumError as e:
+ logger.error("Checksum mismatch for buildtools installer. "
+ "Verify the URL and downloaded file.\n%s" % str(e))
+ sys.exit(1)
+ except bb.fetch2.FetchError as e:
+ logger.error("Failed to download buildtools installer: %s" % str(e))
+ sys.exit(1)
+
+ # Unpack into the standard buildtools download location
+ os.makedirs(buildtools_download_dir, exist_ok=True)
+ fetcher.unpack(buildtools_download_dir)
+
+ filename = os.path.basename(url.rstrip('/'))
+ matches = glob.glob(os.path.join(buildtools_download_dir, '**', filename), recursive=True)
+ if not matches:
+ logger.error("Unpacked file not found: %s" % filename)
+ sys.exit(1)
+ local_path = matches[0]
+
+ logger.plain("Buildtools installer will be installed into {}".format(
+ buildtools_install_dir))
+
+ cmd = [
+ install_buildtools_script,
+ '--local-file', local_path,
+ '-d', buildtools_install_dir,
+ ]
+
+ try:
+ subprocess.check_call(cmd)
+ except subprocess.CalledProcessError as e:
+ logger.error("install-buildtools failed with exit code %d" % e.returncode)
+ sys.exit(1)
def create_siteconf(top_dir, non_interactive, settings):
siteconfpath = os.path.join(top_dir, 'site.conf')
@@ -1286,6 +1363,9 @@ def main():
parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
add_setup_dir_arg(parser_install_buildtools)
parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
+ parser_install_buildtools.add_argument('--default', action='store_true', help='Use install-buildtools script defaults, ignoring any config.')
+ parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.')
+ parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.')
parser_install_buildtools.set_defaults(func=install_buildtools)
parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
index aa546c30b..e1701555e 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -660,6 +660,30 @@ In addition, the command can take the following arguments:
- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
required if :term:`BBPATH` is already configured.
+- ``--url``: full URL to the buildtools SDK installer. Overrides the value
+ from the configuration file.
+
+- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the
+ value from the configuration file.
+
+When ``url`` and ``sha256sum`` are set in the :term:`Configuration File`,
+the installer is downloaded via ``bb.fetch`` (cached in ``DL_DIR``) and its
+checksum is enforced. If no configuration is present, the script falls back
+to its built-in defaults::
+
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "my-config",
+ "install-buildtools": {
+ "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
+ "sha256sum": "af76648b..."
+ },
+ ...
+ }
+ ]
+ }
+
.. _ref-bbsetup-command-settings:
``bitbake-setup settings``
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index 5592e8196..8ca64cb50 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -61,16 +61,31 @@ import getopt
import sys
import os
-opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory="])
+opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "local-file="])
+installdir = None
+local_file = None
for option, value in opts:
if option == '-d':
installdir = value
+ elif option == '--local-file':
+ local_file = value
+ print("install-buildtools local-file={}".format(value))
print("Buildtools installed into {}".format(installdir))
-os.makedirs(installdir)
+os.makedirs(installdir, exist_ok=True)
"""
self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
+ # Dummy buildtools installer for bb.fetch testing
+ self.buildtools_dir = os.path.join(self.tempdir, "buildtools-dl")
+ os.makedirs(self.buildtools_dir)
+ self.buildtools_filename = "x86_64-buildtools-nativesdk-standalone-test.sh"
+ buildtools_filepath = os.path.join(self.buildtools_dir, self.buildtools_filename)
+ with open(buildtools_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho dummy\n")
+ with open(buildtools_filepath, 'rb') as f:
+ self.buildtools_sha256 = hashlib.sha256(f.read()).hexdigest()
+
bitbakeconfigbuild = """#!/usr/bin/env python3
import os
import sys
@@ -108,7 +123,11 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
"name": "gadget",
"description": "Gadget configuration",
"oe-template": "test-configuration-gadget",
- "oe-fragments": ["test-fragment-1"]
+ "oe-fragments": ["test-fragment-1"],
+ "install-buildtools": {
+ "url": "file://%s/%s",
+ "sha256sum": "%s"
+ }
},
{
"name": "gizmo",
@@ -176,7 +195,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
},
"version": "1.0"
}
-""" % (sources)
+""" % (sources, self.buildtools_dir, self.buildtools_filename, self.buildtools_sha256)
os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True)
with open(os.path.join(self.registrypath, name), 'w') as f:
f.write(config)
@@ -644,6 +663,56 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
def _count_layer_backups(self, layers_path):
return len([f for f in os.listdir(layers_path) if 'backup' in f])
+ def test_install_buildtools_fetch(self):
+ """Test that install-buildtools uses bb.fetch with sha256 and passes --local-file"""
+ import shutil
+
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ registry_uri = "git://{};protocol=file;branch=master;rev=master".format(
+ self.registrypath)
+ self.runbbsetup(["settings", "set", "default", "registry", registry_uri])
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
+ self.runbbsetup(["init", "--non-interactive", "test-config-bt", "gadget"])
+ setuppath = self.get_setup_path('test-config-bt', 'gadget')
+
+ # test config-driven install (url + sha256sum from config)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
+
+ # test CLI overrides (use a different file to prove precedence)
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ alt_filename = "alt-buildtools-test.sh"
+ alt_filepath = os.path.join(self.buildtools_dir, alt_filename)
+ with open(alt_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho alt\n")
+ with open(alt_filepath, 'rb') as f:
+ alt_sha256 = hashlib.sha256(f.read()).hexdigest()
+ alt_url = "file://{}/{}".format(self.buildtools_dir, alt_filename)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
+ "--url", alt_url,
+ "--sha256", alt_sha256])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertIn(alt_filename, out[0])
+
+ # test missing sha256sum is a hard error
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'), ignore_errors=True)
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+ config_upstream = json.load(f)
+ config_upstream['bitbake-config']['install-buildtools'] = {
+ 'url': 'file://{}/{}'.format(self.buildtools_dir, self.buildtools_filename)
+ }
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+ json.dump(config_upstream, f)
+ with self.assertRaises(bb.process.ExecutionError):
+ self.runbbsetup(["install-buildtools", "--setup-dir", setuppath, "--force"])
+
def test_update_rebase_conflicts_strategy(self):
"""Test the --rebase-conflicts-strategy option for the update command.
diff --git a/setup-schema/bitbake-setup.schema.json b/setup-schema/bitbake-setup.schema.json
index 177a8dc8c..bb10a9255 100644
--- a/setup-schema/bitbake-setup.schema.json
+++ b/setup-schema/bitbake-setup.schema.json
@@ -170,7 +170,28 @@
},
"setup-dir-name": {
"type": "string",
- "description": "A suggestion for the setup directory name, $-prefixed keys from oe-fragments-one-of will be substituted with user selections"
+ "description": "A suggestion for the setup directory name, $-prefixed keys from oe-fragments-one-of will be substituted with user selections."
+ },
+ "install-buildtools": {
+ "type": "object",
+ "description": "Settings for downloading buildtools via bb.fetch",
+ "properties": {
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Full URL to the buildtools SDK installer."
+ },
+ "sha256sum": {
+ "type": "string",
+ "minLength": 1,
+ "description": "SHA256 checksum of the buildtools installer."
+ }
+ },
+ "required": [
+ "url",
+ "sha256sum"
+ ],
+ "additionalProperties": false
}
},
"additionalProperties": false
--
2.34.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v5] bitbake-setup: use bb.fetch for buildtools installer download
2026-06-30 12:02 ` [PATCH v5] " Jaipaul Cheernam
@ 2026-07-08 15:16 ` Jaipaul Cheernam
2026-07-08 15:26 ` [bitbake-devel] " Alexander Kanavin
0 siblings, 1 reply; 19+ messages in thread
From: Jaipaul Cheernam @ 2026-07-08 15:16 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 78 bytes --]
Hi ,
Could you please share feedback on my patch ?
Regards,
Jaipaul
[-- Attachment #2: Type: text/html, Size: 172 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [bitbake-devel] [PATCH v5] bitbake-setup: use bb.fetch for buildtools installer download
2026-07-08 15:16 ` Jaipaul Cheernam
@ 2026-07-08 15:26 ` Alexander Kanavin
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Kanavin @ 2026-07-08 15:26 UTC (permalink / raw)
To: jaipaul.cheernam; +Cc: bitbake-devel
On Wed, 8 Jul 2026 at 17:16, Jaipaul Cheernam via
lists.openembedded.org
<jaipaul.cheernam=est.tech@lists.openembedded.org> wrote:
> Could you please share feedback on my patch ?
It's in master-next and due for review. If everything is okay, it will
be quietly merged.
Alex
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2026-07-08 15:26 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-15 12:18 [PATCH] bitbake-setup: pass install-buildtools options from config and CLI Jaipaul Cheernam
2026-06-15 13:31 ` [bitbake-devel] " Alexander Kanavin
2026-06-15 14:59 ` Jaipaul Cheernam
2026-06-15 15:38 ` Alexander Kanavin
2026-06-16 10:06 ` Richard Purdie
2026-06-16 14:18 ` Jaipaul Cheernam
2026-06-16 14:26 ` Alexander Kanavin
2026-06-17 10:30 ` Jaipaul Cheernam
2026-06-17 13:28 ` Alexander Kanavin
2026-06-26 13:43 ` [PATCH v2] bitbake-setup: use bb.fetch for buildtools installer download Jaipaul Cheernam
2026-06-26 16:48 ` [bitbake-devel] " Alexander Kanavin
2026-06-26 19:05 ` Jaipaul Cheernam
2026-06-26 19:23 ` [PATCH v3] " Jaipaul Cheernam
2026-06-26 19:44 ` [bitbake-devel] " Alexander Kanavin
2026-06-29 20:34 ` [PATCH v4] " Jaipaul Cheernam
2026-06-30 11:47 ` [bitbake-devel] " Alexander Kanavin
2026-06-30 12:02 ` [PATCH v5] " Jaipaul Cheernam
2026-07-08 15:16 ` Jaipaul Cheernam
2026-07-08 15:26 ` [bitbake-devel] " Alexander Kanavin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.