[PATCH 1/4] setup-schema/layers.schema.json: correct indentation

[PATCH 1/4] setup-schema/layers.schema.json: correct indentation

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

Shift one of the inner blocks 8 spaces to the left, so that
everything is indented by 4 spaces.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 setup-schema/layers.schema.json | 72 ++++++++++++++++-----------------
 1 file changed, 36 insertions(+), 36 deletions(-)

diff --git a/setup-schema/layers.schema.json b/setup-schema/layers.schema.json
index 144ea6d9f..9a0b4ed61 100644
--- a/setup-schema/layers.schema.json
+++ b/setup-schema/layers.schema.json
@@ -27,44 +27,44 @@
                         "type": "boolean"
                     },
                     "git-remote": {
-                                "description": "A remote git source from which to fetch",
+                        "description": "A remote git source from which to fetch",
+                        "type": "object",
+                        "additionalProperties": false,
+                        "required": [
+                            "rev"
+                        ],
+                        "properties": {
+                            "branch": {
+                                "description": "The git branch to fetch (optional)",
+                                "type": "string"
+                            },
+                            "rev": {
+                                "description": "The git revision to checkout",
+                                "type": "string"
+                            },
+                            "describe": {
+                                "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).",
+                                "type": "string"
+                            },
+                            "remotes": {
+                                "description": "The dict of git remotes to add to this repository",
                                 "type": "object",
-                                "additionalProperties": false,
-                                "required": [
-                                    "rev"
-                                ],
-                                "properties": {
-                                    "branch": {
-                                        "description": "The git branch to fetch (optional)",
-                                        "type": "string"
-                                    },
-                                    "rev": {
-                                        "description": "The git revision to checkout",
-                                        "type": "string"
-                                    },
-                                    "describe": {
-                                        "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).",
-                                        "type": "string"
-                                    },
-                                    "remotes": {
-                                        "description": "The dict of git remotes to add to this repository",
-                                        "type": "object",
-                                        "patternProperties": { ".*" : {
-                                            "description": "A git remote",
-                                            "type": "object",
-                                            "addtionalProperties": false,
-                                            "required": [
-                                                "uri"
-                                            ],
-                                            "properties": {
-                                                "uri": {
-                                                    "description": "The URI for the remote, using git URL syntax",
-                                                    "type": "string"
-                                                }
-                                            }
-                                        }}
+                                "patternProperties": { ".*" : {
+                                    "description": "A git remote",
+                                    "type": "object",
+                                    "addtionalProperties": false,
+                                    "required": [
+                                        "uri"
+                                    ],
+                                    "properties": {
+                                        "uri": {
+                                            "description": "The URI for the remote, using git URL syntax",
+                                            "type": "string"
+                                        }
                                     }
-                                }
+                                }}
+                            }
+                        }
                     }
                 }
             }
-- 
2.47.3

[PATCH 2/4] bitbake-setup: override complete source entries, not just the git-remote properties

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

The code was assuming that a source entry always contains
a git-remote property, and that is the only property that
should be replaced.

With upcoming introduction of local sources (using 'local' as a
property as an alternative to the existing 'git-remote'
property) that is no longer true.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 bin/bitbake-setup | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index b1d751899..8d4e9769a 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -323,7 +323,7 @@ def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt"
     layer_overrides = config["source-overrides"]["sources"]
     for k,v in layer_overrides.items():
         if k in layer_config:
-            layer_config[k]["git-remote"] = v["git-remote"]
+            layer_config[k] = v
     sources_fixed_revisions = checkout_layers(layer_config, layerdir, d)
     bitbake_config = config["bitbake-config"]
     thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None
-- 
2.47.3

[PATCH 3/4] bitbake-setup: use separate functions for git-specific operations

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

This prepares the code for adding different type of sources than git remotes:

- put git-specific operations into their own functions (no behavior changes)

- call them only if a git-remote entry actually exists in the configuration;
do not assume it is always there.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 bin/bitbake-setup | 42 ++++++++++++++++++++++++++----------------
 1 file changed, 26 insertions(+), 16 deletions(-)

diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 8d4e9769a..1f83b1b2a 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -90,16 +90,7 @@ def _write_layer_list(dest, repodirs):
         json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4)
 
 def checkout_layers(layers, layerdir, d):
-    layers_fixed_revisions = copy.deepcopy(layers)
-    repodirs = []
-    oesetupbuild = None
-    print("Fetching layer/tool repositories into {}".format(layerdir))
-    for r_name in layers:
-        r_data = layers[r_name]
-        repodir = r_data.get("path", r_name)
-        repodirs.append(repodir)
-
-        r_remote = r_data['git-remote']
+    def _checkout_git_remote(r_remote, repodir, layers_fixed_revisions):
         rev = r_remote['rev']
         branch = r_remote.get('branch', None)
         remotes = r_remote['remotes']
@@ -118,6 +109,19 @@ def checkout_layers(layers, layerdir, d):
             revision = urldata.revision
             layers_fixed_revisions[r_name]['git-remote']['rev'] = revision
 
+    layers_fixed_revisions = copy.deepcopy(layers)
+    repodirs = []
+    oesetupbuild = None
+    print("Fetching layer/tool repositories into {}".format(layerdir))
+    for r_name in layers:
+        r_data = layers[r_name]
+        repodir = r_data.get("path", r_name)
+        repodirs.append(repodir)
+
+        r_remote = r_data.get('git-remote')
+        if r_remote:
+            _checkout_git_remote(r_remote, repodir, layers_fixed_revisions)
+
         if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')):
             oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build')
             oeinitbuildenvdir = os.path.join(layerdir, repodir)
@@ -563,12 +567,8 @@ def get_diff(file1, file2):
     return None
 
 def are_layers_changed(layers, layerdir, d):
-    changed = False
-    for r_name in layers:
-        r_data = layers[r_name]
-        repodir = r_data.get("path", r_name)
-
-        r_remote = r_data['git-remote']
+    def _is_git_remote_changed(r_remote, repodir):
+        changed = False
         rev = r_remote['rev']
         branch = r_remote.get('branch', None)
         remotes = r_remote['remotes']
@@ -586,6 +586,16 @@ def are_layers_changed(layers, layerdir, d):
             if upstream_revision != local_revision:
                 changed = True
                 print('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remotes[remote]["uri"], os.path.join(layerdir, repodir), rev, local_revision, upstream_revision))
+        return changed
+
+    changed = False
+    for r_name in layers:
+        r_data = layers[r_name]
+        repodir = r_data.get("path", r_name)
+
+        git_remote = r_data.get('git-remote', repodir)
+        if git_remote:
+            changed = changed | _is_git_remote_changed(git_remote, repodir)
 
     return changed
 
-- 
2.47.3

[PATCH 4/4] bitbake-setup: implement symlinking local sources into builds

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

The feature and the use case were proposed here:
https://lists.openembedded.org/g/bitbake-devel/message/18373

The implementation extends the schema with a 'local' source type,
and simply symlinks them into the setup directory during 'init'.
'status' or 'update' do not consider or modify the symlinks.

The overrides support is extended to add a command line 'shortcut'
for overriding sources with a local path, and massaging relative or
~-containing paths as appropriate.

Documentation is extended to describe the local sources and show
examples. The json properties for sources are also grouped correctly to show
what is git-specific, what is local-specific and what is common.

Tests are extended to cover a few things that weren't previously tested:
source overrides, custom setup directory names, and tests for local
sources are added as well.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 bin/bitbake-setup                             | 29 ++++++++++-
 .../bitbake-user-manual-environment-setup.rst | 50 +++++++++++++++++--
 lib/bb/tests/setup.py                         | 28 +++++++++++
 setup-schema/layers.schema.json               | 14 ++++++
 4 files changed, 115 insertions(+), 6 deletions(-)

diff --git a/bin/bitbake-setup b/bin/bitbake-setup
index 1f83b1b2a..30c5c44f7 100755
--- a/bin/bitbake-setup
+++ b/bin/bitbake-setup
@@ -109,6 +109,10 @@ def checkout_layers(layers, layerdir, d):
             revision = urldata.revision
             layers_fixed_revisions[r_name]['git-remote']['rev'] = revision
 
+    def _symlink_local(src, dst):
+        print("Making a symbolic link {} pointing to {}".format(dst, src))
+        os.symlink(src, dst)
+
     layers_fixed_revisions = copy.deepcopy(layers)
     repodirs = []
     oesetupbuild = None
@@ -121,6 +125,9 @@ def checkout_layers(layers, layerdir, d):
         r_remote = r_data.get('git-remote')
         if r_remote:
             _checkout_git_remote(r_remote, repodir, layers_fixed_revisions)
+        local = r_data.get('local')
+        if local:
+            _symlink_local(os.path.expanduser(local["path"]), os.path.join(layerdir,repodir))
 
         if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')):
             oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build')
@@ -485,6 +492,24 @@ def obtain_config(top_dir, settings, args, source_overrides, d):
     upstream_config['skip-selection'] = args.skip_selection
     return upstream_config
 
+def obtain_overrides(args):
+    overrides = {'sources':{}}
+    if args.source_overrides:
+        overrides = json.load(open(args.source_overrides))
+        overrides_dir = os.path.dirname(os.path.abspath(args.source_overrides))
+        for s,v in overrides['sources'].items():
+            local = v.get('local')
+            if local:
+                path = os.path.expanduser(local['path'])
+                if not os.path.isabs(path):
+                    overrides['sources'][s]['local']['path'] = os.path.join(overrides_dir, path)
+
+    for local_name, local_path in args.use_local_source:
+        overrides['sources'][local_name] = {'local':{'path':os.path.abspath(os.path.expanduser(local_path))}}
+
+    return overrides
+
+
 def init_config(top_dir, settings, args):
     create_siteconf(top_dir, args.non_interactive, settings)
 
@@ -495,7 +520,7 @@ def init_config(top_dir, settings, args):
         progress = event.progress if event.progress > 0 else 0
         print("{}% {}                ".format(progress, rate), file=stdout, end='\r')
 
-    source_overrides = json.load(open(args.source_overrides)) if args.source_overrides else {'sources':{}}
+    source_overrides = obtain_overrides(args)
     upstream_config = obtain_config(top_dir, settings, args, source_overrides, d)
     print("\nRun 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options'])))
 
@@ -904,6 +929,8 @@ def main():
     parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.')
     parser_init.add_argument('--setup-dir-name', action='store', help='A custom setup directory name under the top directory.')
     parser_init.add_argument('--skip-selection', action='append', help='Do not select and set an option/fragment from available choices; the resulting bitbake configuration may be incomplete.')
+    parser_init.add_argument('-L', '--use-local-source', default=[], action='append', nargs=2, metavar=('SOURCE_NAME', 'PATH'),
+                        help='Symlink local source into a build, instead of getting it as prescribed by a configuration (useful for local development).')
     parser_init.set_defaults(func=init_config)
 
     parser_status = subparsers.add_parser('status', help='Check if the setup needs to be synchronized with configuration')
diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
index 12b29241f..4d3585f09 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -317,6 +317,17 @@ In addition, the command can take the following arguments:
 -  ``--skip-selection``: can be used to skip some of the choices
    (which may result in an incomplete :term:`Setup`!)
 
+-  ``-L`` or ``--use-local-source``: instead of getting a source as prescribed in
+   a configuration, symlink it into a :term:`Setup` from a path on local disk. This
+   is useful for local development where that particular source directory is managed
+   separately, and bitbake-setup will include it in a build but will not otherwise
+   touch or modify it. This option can be specified multiple times to specify multiple
+   local sources.
+
+   The option can be seen as a command line shortcut to providing an override file
+   with a ``local`` source in it. See the :ref:`ref-bbsetup-source-overrides` section
+   for more information on source overrides
+
 ``bitbake-setup init`` Examples
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -649,7 +660,8 @@ They contain the following sections:
           "description": "OpenEmbedded - 'nodistro' basic configuration"
       }
 
--  ``sources`` (*optional*): Git repositories to fetch.
+-  ``sources`` (*optional*): sources, such as git repositories that should be provided
+   under ``layers/`` directory of a :term:`Setup`.
 
    Example:
 
@@ -669,12 +681,31 @@ They contain the following sections:
                      "rev": "master"
                  },
                  "path": "bitbake"
+             },
+             "openembedded-core": {
+                 "local": {
+                     "path": "~/path/to/local/openembedded-core"
+                 }
              }
          }
       }
 
    Sources can be specified with the following options:
 
+   -  ``path`` (*optional*): where the source is extracted, relative to the
+      ``layers/`` directory of a :term:`Setup`. If unspecified, the name of the
+      source is used.
+
+   -  ``git-remote`` (*optional*): specifies URI, branch and revision of a git
+      repository to fetch from.
+
+   -  ``local`` (*optional*): specifies a path on local disk that should be symlinked
+      to under ``layers\``. This is useful for local development, where some layer
+      or other component used in a build is managed separately, but should still be
+      available for bitbake-setup driven builds.
+
+   ``git-remote`` entries are specified with the following options:
+
    -  ``uri`` (**required**): a URI that follows the git URI syntax.
       See https://git-scm.com/docs/git-clone#_git_urls for more information.
 
@@ -688,9 +719,13 @@ They contain the following sections:
    -  ``branch`` (**required**): the Git branch, used to check that the
       specified ``rev`` is indeed on that branch.
 
-   -  ``path`` (*optional*): where the source is extracted, relative to the
-      ``layers/`` directory of a :term:`Setup`. If unspecified, the name of the
-      source is used.
+   ``local`` entries are specified with the following options:
+
+   -  ``path`` (**required**): the path on local disk where the externally
+      managed source tree is. ``~`` and ``~user`` are expanded to that user's home
+      directory. Paths in configuration files must be absolute (after possible
+      ~ expansion), paths in override files can be relative to the directory where
+      the override file is.
 
 -  ``expires`` (*optional*): Expiration date of the configuration. This date
    should be in :wikipedia:`ISO 8601 <ISO_8601>` format (``YYYY-MM-DDTHH:MM:SS``).
@@ -868,7 +903,7 @@ The ``--source-overrides`` option can be passed multiple times, in which case th
 overrides are applied in the order specified in the command-line.
 
 Here is an example file that overrides the branch of the BitBake repository to
-"master-next":
+"master-next", and provides openembedded-core as a symlink to a path on local disk:
 
 .. code-block:: json
 
@@ -885,6 +920,11 @@ Here is an example file that overrides the branch of the BitBake repository to
                    },
                    "rev": "master-next"
                }
+           },
+           "openembedded-core": {
+               "local": {
+                   "path": "~/path/to/local/openembedded-core"
+               }
            }
        },
        "version": "1.0"
diff --git a/lib/bb/tests/setup.py b/lib/bb/tests/setup.py
index 8b6d8bce6..933178c84 100644
--- a/lib/bb/tests/setup.py
+++ b/lib/bb/tests/setup.py
@@ -320,6 +320,34 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"]))
                 out = self.runbbsetup("update --update-bb-conf='yes'")
                 self.assertIn("Configuration in {} has not changed".format(setuppath), out[0])
 
+        # check source overrides, local sources provided with symlinks, and custom setup dir name
+        source_override_content = """
+{
+    "sources": {
+        "test-repo": {
+            "local": {
+                "path": "."
+            }
+        }
+    }
+}"""
+        override_filename = 'source-overrides.json'
+        custom_setup_dir = 'special-setup-dir'
+        self.add_file_to_testrepo(override_filename, source_override_content)
+        out = self.runbbsetup("init --non-interactive --source-overrides {} --setup-dir-name {} test-config-1 gadget".format(os.path.join(self.testrepopath, override_filename), custom_setup_dir))
+        custom_setup_path = os.path.join(self.tempdir, 'bitbake-builds', custom_setup_dir)
+        custom_layer_path = os.path.join(custom_setup_path, 'layers', 'test-repo')
+        self.assertTrue(os.path.islink(custom_layer_path))
+        self.assertEqual(self.testrepopath, os.path.realpath(custom_layer_path))
+
+        # same but use command line options to specify local overrides
+        custom_setup_dir = 'special-setup-dir-with-cmdline-overrides'
+        out = self.runbbsetup("init --non-interactive -L test-repo {} --setup-dir-name {} test-config-1 gadget".format(self.testrepopath, custom_setup_dir))
+        custom_setup_path = os.path.join(self.tempdir, 'bitbake-builds', custom_setup_dir)
+        custom_layer_path = os.path.join(custom_setup_path, 'layers', 'test-repo')
+        self.assertTrue(os.path.islink(custom_layer_path))
+        self.assertEqual(self.testrepopath, os.path.realpath(custom_layer_path))
+
         # install buildtools
         out = self.runbbsetup("install-buildtools")
         self.assertIn("Buildtools installed into", out[0])
diff --git a/setup-schema/layers.schema.json b/setup-schema/layers.schema.json
index 9a0b4ed61..1a0255435 100644
--- a/setup-schema/layers.schema.json
+++ b/setup-schema/layers.schema.json
@@ -65,6 +65,20 @@
                                 }}
                             }
                         }
+                    },
+                    "local": {
+                        "description": "A local directory that should be made available for builds via symlinking",
+                        "type": "object",
+                        "additionalProperties": false,
+                        "required": [
+                            "path"
+                        ],
+                        "properties": {
+                            "path": {
+                                "description": "The path to the directory",
+                                "type": "string"
+                            }
+                        }
                     }
                 }
             }
-- 
2.47.3

Re: [bitbake-devel] [PATCH 4/4] bitbake-setup: implement symlinking local sources into builds

[Antonin Godard]

Hi,

On Thu Dec 11, 2025 at 3:25 PM CET, Alexander Kanavin via lists.openembedded.org wrote:
[...]
> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> index 12b29241f..4d3585f09 100644
> --- a/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> +++ b/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
> @@ -317,6 +317,17 @@ In addition, the command can take the following arguments:
>  -  ``--skip-selection``: can be used to skip some of the choices
>     (which may result in an incomplete :term:`Setup`!)
>  
> +-  ``-L`` or ``--use-local-source``: instead of getting a source as prescribed in
> +   a configuration, symlink it into a :term:`Setup` from a path on local disk. This
> +   is useful for local development where that particular source directory is managed
> +   separately, and bitbake-setup will include it in a build but will not otherwise
> +   touch or modify it. This option can be specified multiple times to specify multiple
> +   local sources.
> +
> +   The option can be seen as a command line shortcut to providing an override file
> +   with a ``local`` source in it. See the :ref:`ref-bbsetup-source-overrides` section
> +   for more information on source overrides

Missing a period.

> +
>  ``bitbake-setup init`` Examples
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Maybe we could add an example (or extend one) in this section to show how this
new --use-local-source option is used?

>  
> @@ -649,7 +660,8 @@ They contain the following sections:
>            "description": "OpenEmbedded - 'nodistro' basic configuration"
>        }
>  
> --  ``sources`` (*optional*): Git repositories to fetch.
> +-  ``sources`` (*optional*): sources, such as git repositories that should be provided
> +   under ``layers/`` directory of a :term:`Setup`.
>  
>     Example:
>  
> @@ -669,12 +681,31 @@ They contain the following sections:
>                       "rev": "master"
>                   },
>                   "path": "bitbake"
> +             },
> +             "openembedded-core": {
> +                 "local": {
> +                     "path": "~/path/to/local/openembedded-core"
> +                 }
>               }
>           }
>        }
>  
>     Sources can be specified with the following options:
>  
> +   -  ``path`` (*optional*): where the source is extracted, relative to the
> +      ``layers/`` directory of a :term:`Setup`. If unspecified, the name of the
> +      source is used.
> +
> +   -  ``git-remote`` (*optional*): specifies URI, branch and revision of a git
> +      repository to fetch from.

I'd maybe move git-remote below local and just make one bullet point for the
git-remote description. Because at the moment it appears to be under "local".

So like that:

"""
-  ``git-remote`` (*optional*): specifies URI, branch and revision of a git
   repository to fetch from.

   ``git-remote`` entries are specified with the following options:

   -  ``uri`` (**required**): a URI that follows the git URI syntax.
      See https://git-scm.com/docs/git-clone#_git_urls for more information.

   ...
"""

> +
> +   -  ``local`` (*optional*): specifies a path on local disk that should be symlinked
> +      to under ``layers\``. This is useful for local development, where some layer

Should be "/" instead of "\"

> +      or other component used in a build is managed separately, but should still be
> +      available for bitbake-setup driven builds.
> +
> +   ``git-remote`` entries are specified with the following options:
> +
>     -  ``uri`` (**required**): a URI that follows the git URI syntax.
>        See https://git-scm.com/docs/git-clone#_git_urls for more information.
>  
> @@ -688,9 +719,13 @@ They contain the following sections:
>     -  ``branch`` (**required**): the Git branch, used to check that the
>        specified ``rev`` is indeed on that branch.
>  
> -   -  ``path`` (*optional*): where the source is extracted, relative to the
> -      ``layers/`` directory of a :term:`Setup`. If unspecified, the name of the
> -      source is used.
> +   ``local`` entries are specified with the following options:
> +
> +   -  ``path`` (**required**): the path on local disk where the externally
> +      managed source tree is. ``~`` and ``~user`` are expanded to that user's home
> +      directory. Paths in configuration files must be absolute (after possible
> +      ~ expansion), paths in override files can be relative to the directory where
> +      the override file is.

Likewise, I would put this under the description of "local", like git-remote. In
the end it will group and makes things a bit easier to follow IMO.

Otherwise LGTM, thanks for the docs update. :)

Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCH 4/4] bitbake-setup: implement symlinking local sources into builds

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 2909 bytes --]

On Thu, 2025-12-11 at 15:25 +0100, Alexander Kanavin wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> The feature and the use case were proposed here:
> https://lists.openembedded.org/g/bitbake-devel/message/18373
> 
> The implementation extends the schema with a 'local' source type,
> and simply symlinks them into the setup directory during 'init'.
> 'status' or 'update' do not consider or modify the symlinks.
> 
> The overrides support is extended to add a command line 'shortcut'
> for overriding sources with a local path, and massaging relative or
> ~-containing paths as appropriate.
> 
> Documentation is extended to describe the local sources and show
> examples. The json properties for sources are also grouped correctly to show
> what is git-specific, what is local-specific and what is common.
> 
> Tests are extended to cover a few things that weren't previously tested:
> source overrides, custom setup directory names, and tests for local
> sources are added as well.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  bin/bitbake-setup                             | 29 ++++++++++-
>  .../bitbake-user-manual-environment-setup.rst | 50 +++++++++++++++++--
>  lib/bb/tests/setup.py                         | 28 +++++++++++
>  setup-schema/layers.schema.json               | 14 ++++++
>  4 files changed, 115 insertions(+), 6 deletions(-)
> 
> diff --git a/bin/bitbake-setup b/bin/bitbake-setup
> index 1f83b1b2a..30c5c44f7 100755
> --- a/bin/bitbake-setup
> +++ b/bin/bitbake-setup
> @@ -109,6 +109,10 @@ def checkout_layers(layers, layerdir, d):
>              revision = urldata.revision
>              layers_fixed_revisions[r_name]['git-remote']['rev'] = revision
>  
> +    def _symlink_local(src, dst):
> +        print("Making a symbolic link {} pointing to {}".format(dst, src))
> +        os.symlink(src, dst)
> +
>      layers_fixed_revisions = copy.deepcopy(layers)
>      repodirs = []
>      oesetupbuild = None
> @@ -121,6 +125,9 @@ def checkout_layers(layers, layerdir, d):
>          r_remote = r_data.get('git-remote')
>          if r_remote:
>              _checkout_git_remote(r_remote, repodir, layers_fixed_revisions)
> +        local = r_data.get('local')
> +        if local:
> +            _symlink_local(os.path.expanduser(local["path"]), os.path.join(layerdir,repodir))

Applying overrides shouldn't result in both 'git-remote' and 'local'
keys being present for the same source, but a badly written .conf.json
file could contain both. We should probably error out if both keys are
provided.

We call os.path.expanduser() in obtain_overrides, so I don't think we
need to call it again here.

And, I'd prefer the variable name r_local to match r_remote.

The rest of this looks great and works well in my local testing. Thanks
for implementing this!

-- 
Paul Barker



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 1/4] setup-schema/layers.schema.json: correct indentation

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 5447 bytes --]

On Thu, 2025-12-11 at 15:25 +0100, Alexander Kanavin wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> Shift one of the inner blocks 8 spaces to the left, so that
> everything is indented by 4 spaces.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  setup-schema/layers.schema.json | 72 ++++++++++++++++-----------------
>  1 file changed, 36 insertions(+), 36 deletions(-)
> 
> diff --git a/setup-schema/layers.schema.json b/setup-schema/layers.schema.json
> index 144ea6d9f..9a0b4ed61 100644
> --- a/setup-schema/layers.schema.json
> +++ b/setup-schema/layers.schema.json
> @@ -27,44 +27,44 @@
>                          "type": "boolean"
>                      },
>                      "git-remote": {
> -                                "description": "A remote git source from which to fetch",
> +                        "description": "A remote git source from which to fetch",
> +                        "type": "object",
> +                        "additionalProperties": false,
> +                        "required": [
> +                            "rev"
> +                        ],
> +                        "properties": {
> +                            "branch": {
> +                                "description": "The git branch to fetch (optional)",
> +                                "type": "string"
> +                            },
> +                            "rev": {
> +                                "description": "The git revision to checkout",
> +                                "type": "string"
> +                            },
> +                            "describe": {
> +                                "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).",
> +                                "type": "string"
> +                            },
> +                            "remotes": {
> +                                "description": "The dict of git remotes to add to this repository",
>                                  "type": "object",
> -                                "additionalProperties": false,
> -                                "required": [
> -                                    "rev"
> -                                ],
> -                                "properties": {
> -                                    "branch": {
> -                                        "description": "The git branch to fetch (optional)",
> -                                        "type": "string"
> -                                    },
> -                                    "rev": {
> -                                        "description": "The git revision to checkout",
> -                                        "type": "string"
> -                                    },
> -                                    "describe": {
> -                                        "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).",
> -                                        "type": "string"
> -                                    },
> -                                    "remotes": {
> -                                        "description": "The dict of git remotes to add to this repository",
> -                                        "type": "object",
> -                                        "patternProperties": { ".*" : {
> -                                            "description": "A git remote",
> -                                            "type": "object",
> -                                            "addtionalProperties": false,
> -                                            "required": [
> -                                                "uri"
> -                                            ],
> -                                            "properties": {
> -                                                "uri": {
> -                                                    "description": "The URI for the remote, using git URL syntax",
> -                                                    "type": "string"
> -                                                }
> -                                            }
> -                                        }}
> +                                "patternProperties": { ".*" : {
> +                                    "description": "A git remote",
> +                                    "type": "object",
> +                                    "addtionalProperties": false,
> +                                    "required": [
> +                                        "uri"
> +                                    ],
> +                                    "properties": {
> +                                        "uri": {
> +                                            "description": "The URI for the remote, using git URL syntax",
> +                                            "type": "string"
> +                                        }
>                                      }
> -                                }
> +                                }}
> +                            }
> +                        }
>                      }
>                  }
>              }

Reviewed-by: Paul Barker <paul@pbarker.dev>

Best regards,

-- 
Paul Barker



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/4] bitbake-setup: override complete source entries, not just the git-remote properties

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

On Thu, 2025-12-11 at 15:25 +0100, Alexander Kanavin wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> The code was assuming that a source entry always contains
> a git-remote property, and that is the only property that
> should be replaced.
> 
> With upcoming introduction of local sources (using 'local' as a
> property as an alternative to the existing 'git-remote'
> property) that is no longer true.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  bin/bitbake-setup | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/bin/bitbake-setup b/bin/bitbake-setup
> index b1d751899..8d4e9769a 100755
> --- a/bin/bitbake-setup
> +++ b/bin/bitbake-setup
> @@ -323,7 +323,7 @@ def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt"
>      layer_overrides = config["source-overrides"]["sources"]
>      for k,v in layer_overrides.items():
>          if k in layer_config:
> -            layer_config[k]["git-remote"] = v["git-remote"]
> +            layer_config[k] = v
>      sources_fixed_revisions = checkout_layers(layer_config, layerdir, d)
>      bitbake_config = config["bitbake-config"]
>      thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None

Reviewed-by: Paul Barker <paul@pbarker.dev>

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 3/4] bitbake-setup: use separate functions for git-specific operations

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 3809 bytes --]

On Thu, 2025-12-11 at 15:25 +0100, Alexander Kanavin wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> This prepares the code for adding different type of sources than git remotes:
> 
> - put git-specific operations into their own functions (no behavior changes)
> 
> - call them only if a git-remote entry actually exists in the configuration;
> do not assume it is always there.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  bin/bitbake-setup | 42 ++++++++++++++++++++++++++----------------
>  1 file changed, 26 insertions(+), 16 deletions(-)
> 
> diff --git a/bin/bitbake-setup b/bin/bitbake-setup
> index 8d4e9769a..1f83b1b2a 100755
> --- a/bin/bitbake-setup
> +++ b/bin/bitbake-setup
> @@ -90,16 +90,7 @@ def _write_layer_list(dest, repodirs):
>          json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4)
>  
>  def checkout_layers(layers, layerdir, d):
> -    layers_fixed_revisions = copy.deepcopy(layers)
> -    repodirs = []
> -    oesetupbuild = None
> -    print("Fetching layer/tool repositories into {}".format(layerdir))
> -    for r_name in layers:
> -        r_data = layers[r_name]
> -        repodir = r_data.get("path", r_name)
> -        repodirs.append(repodir)
> -
> -        r_remote = r_data['git-remote']
> +    def _checkout_git_remote(r_remote, repodir, layers_fixed_revisions):
>          rev = r_remote['rev']
>          branch = r_remote.get('branch', None)
>          remotes = r_remote['remotes']
> @@ -118,6 +109,19 @@ def checkout_layers(layers, layerdir, d):
>              revision = urldata.revision
>              layers_fixed_revisions[r_name]['git-remote']['rev'] = revision
>  
> +    layers_fixed_revisions = copy.deepcopy(layers)
> +    repodirs = []
> +    oesetupbuild = None
> +    print("Fetching layer/tool repositories into {}".format(layerdir))
> +    for r_name in layers:
> +        r_data = layers[r_name]
> +        repodir = r_data.get("path", r_name)
> +        repodirs.append(repodir)
> +
> +        r_remote = r_data.get('git-remote')
> +        if r_remote:
> +            _checkout_git_remote(r_remote, repodir, layers_fixed_revisions)
> +
>          if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')):
>              oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build')
>              oeinitbuildenvdir = os.path.join(layerdir, repodir)
> @@ -563,12 +567,8 @@ def get_diff(file1, file2):
>      return None
>  
>  def are_layers_changed(layers, layerdir, d):
> -    changed = False
> -    for r_name in layers:
> -        r_data = layers[r_name]
> -        repodir = r_data.get("path", r_name)
> -
> -        r_remote = r_data['git-remote']
> +    def _is_git_remote_changed(r_remote, repodir):
> +        changed = False
>          rev = r_remote['rev']
>          branch = r_remote.get('branch', None)
>          remotes = r_remote['remotes']
> @@ -586,6 +586,16 @@ def are_layers_changed(layers, layerdir, d):
>              if upstream_revision != local_revision:
>                  changed = True
>                  print('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remotes[remote]["uri"], os.path.join(layerdir, repodir), rev, local_revision, upstream_revision))
> +        return changed
> +
> +    changed = False
> +    for r_name in layers:
> +        r_data = layers[r_name]
> +        repodir = r_data.get("path", r_name)
> +
> +        git_remote = r_data.get('git-remote', repodir)
> +        if git_remote:
> +            changed = changed | _is_git_remote_changed(git_remote, repodir)
>  
>      return changed
>  

Reviewed-by: Paul Barker <paul@pbarker.dev>

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 4/4] bitbake-setup: implement symlinking local sources into builds

[Alexander Kanavin]

On Fri, 12 Dec 2025 at 12:27, Paul Barker <paul@pbarker.dev> wrote:

> Applying overrides shouldn't result in both 'git-remote' and 'local'
> keys being present for the same source, but a badly written .conf.json
> file could contain both. We should probably error out if both keys are
> provided.

Yes, I'll add a check for this.

> We call os.path.expanduser() in obtain_overrides, so I don't think we
> need to call it again here.

This is for the situation where there are no overrides, and the base
configuration itself contains a local source with ~ in it - e.g.
something that is intended for private use on a specific machine.

> And, I'd prefer the variable name r_local to match r_remote.

That will be corrected as well.

Alex

Re: [bitbake-devel] [PATCH 4/4] bitbake-setup: implement symlinking local sources into builds

[Alexander Kanavin]

On Fri, 12 Dec 2025 at 09:11, Antonin Godard <antonin.godard@bootlin.com> wrote:

> Likewise, I would put this under the description of "local", like git-remote. In
> the end it will group and makes things a bit easier to follow IMO.
>
> Otherwise LGTM, thanks for the docs update. :)

Thanks, I'll address these all in v2.

Alex