* [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace
2026-05-22 7:59 [PATCH v2 0/3] devtool: fix standalone clone conversion for nested git repos Jamin Lin
@ 2026-05-22 7:59 ` Jamin Lin
2026-05-22 10:41 ` Paul Barker
2026-05-22 7:59 ` [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe Jamin Lin
2026-05-22 7:59 ` [PATCH v2 3/3] oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos Jamin Lin
2 siblings, 1 reply; 6+ messages in thread
From: Jamin Lin @ 2026-05-22 7:59 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org, alex.kanavin@gmail.com,
paul@pbarker.dev
Cc: Troy Lee, Jamin Lin, Vince Chang
When a recipe uses multiple git SRC_URI entries with different destsuffix
values (e.g. Zephyr-based recipes with separate repositories for the
kernel, modules and application), do_unpack clones each source tree with
'git clone -n -s'.
The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file
pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
git_convert_standalone_clone() is called by devtool_post_unpack to make
the workspace standalone: it runs 'git repack -a' to copy all objects
into the local object store and then removes the alternates file.
However it only processes the top-level source directory. Each nested
git repo created by a separate SRC_URI entry retains its own alternates
file still pointing into downloads/.
Steps to reproduce:
1. devtool modify <recipe-with-multiple-git-SRC_URI>
2. bitbake -c cleanall <recipe>
3. bitbake <recipe>
At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
the bare repositories from downloads/git2/. The top-level workspace
repo is standalone (alternates already removed by the original code),
but the nested repos still hold alternates pointing to the now-deleted
paths.
At step 3, srctree_hash_files() runs 'git add -A .' with a custom
GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
each nested repo to check for changes; this fails with exit 128 because
the nested alternates are broken:
error: unable to normalize alternate object path:
.../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
fatal: bad object HEAD
fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica
This halts the BitBake parse phase with a CalledProcessError and leaves
the workspace in an unrecoverable state without manual intervention.
Fix by extending git_convert_standalone_clone() to walk the source tree
and apply the same 'git repack -a' + remove-alternates treatment to
every nested git repository found, making the entire workspace fully
standalone at devtool modify time. The walk uses dirs[:] = [] to stop
at each git boundary so it never descends into already-converted repos.
Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
1 file changed, 24 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 32e749dbb1..0a83470373 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
def git_convert_standalone_clone(repodir):
- """If specified directory is a git repository, ensure it's a standalone clone"""
+ """
+ If specified directory is a git repository, ensure it's a standalone clone.
+ Also converts any nested git repositories (created by multiple SRC_URI git
+ entries with different destsuffix values) so that none of their contents
+ depend on the shared downloads directory via alternates.
+ """
+
import bb.process
- if os.path.exists(os.path.join(repodir, '.git')):
- alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
+
+ def _convert(gitdir, workdir):
+ alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
if os.path.exists(alternatesfile):
# This will have been cloned with -s, so we need to convert it so none
# of the contents is shared
- bb.process.run('git repack -a', cwd=repodir)
+ bb.process.run('git repack -a', cwd=workdir)
os.remove(alternatesfile)
+ if os.path.exists(os.path.join(repodir, '.git')):
+ _convert(os.path.join(repodir, '.git'), repodir)
+
+ # Also handle nested git repos created by multiple SRC_URI git entries
+ # with different destsuffix values. Each nested repo is cloned with -s
+ # and has its own alternates pointing to the downloads directory.
+ for root, dirs, files in os.walk(repodir):
+ if root == repodir:
+ continue
+ if '.git' in dirs:
+ _convert(os.path.join(root, '.git'), root)
+ dirs[:] = [] # don't recurse into nested repos
+
def _get_temp_recipe_dir(d):
# This is a little bit hacky but we need to find a place where we can put
# the recipe so that bitbake can find it. We're going to delete it at the
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace
2026-05-22 7:59 ` [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace Jamin Lin
@ 2026-05-22 10:41 ` Paul Barker
0 siblings, 0 replies; 6+ messages in thread
From: Paul Barker @ 2026-05-22 10:41 UTC (permalink / raw)
To: Jamin Lin, openembedded-core@lists.openembedded.org,
alex.kanavin@gmail.com
Cc: Troy Lee, Vince Chang
[-- Attachment #1: Type: text/plain, Size: 5088 bytes --]
On Fri, 2026-05-22 at 07:59 +0000, Jamin Lin wrote:
> When a recipe uses multiple git SRC_URI entries with different destsuffix
> values (e.g. Zephyr-based recipes with separate repositories for the
> kernel, modules and application), do_unpack clones each source tree with
> 'git clone -n -s'.
>
> The -s flag uses git's shared-object mechanism:
> instead of copying objects locally it writes a .git/objects/info/alternates file
> pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
>
> git_convert_standalone_clone() is called by devtool_post_unpack to make
> the workspace standalone: it runs 'git repack -a' to copy all objects
> into the local object store and then removes the alternates file.
>
> However it only processes the top-level source directory. Each nested
> git repo created by a separate SRC_URI entry retains its own alternates
> file still pointing into downloads/.
>
> Steps to reproduce:
> 1. devtool modify <recipe-with-multiple-git-SRC_URI>
> 2. bitbake -c cleanall <recipe>
> 3. bitbake <recipe>
>
> At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
> the bare repositories from downloads/git2/. The top-level workspace
> repo is standalone (alternates already removed by the original code),
> but the nested repos still hold alternates pointing to the now-deleted
> paths.
>
> At step 3, srctree_hash_files() runs 'git add -A .' with a custom
> GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
> each nested repo to check for changes; this fails with exit 128 because
> the nested alternates are broken:
> error: unable to normalize alternate object path:
> .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
> fatal: bad object HEAD
> fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica
>
> This halts the BitBake parse phase with a CalledProcessError and leaves
> the workspace in an unrecoverable state without manual intervention.
>
> Fix by extending git_convert_standalone_clone() to walk the source tree
> and apply the same 'git repack -a' + remove-alternates treatment to
> every nested git repository found, making the entire workspace fully
> standalone at devtool modify time. The walk uses dirs[:] = [] to stop
> at each git boundary so it never descends into already-converted repos.
>
> Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
> ---
> scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
> 1 file changed, 24 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
> index 32e749dbb1..0a83470373 100644
> --- a/scripts/lib/scriptutils.py
> +++ b/scripts/lib/scriptutils.py
> @@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
>
>
> def git_convert_standalone_clone(repodir):
> - """If specified directory is a git repository, ensure it's a standalone clone"""
> + """
> + If specified directory is a git repository, ensure it's a standalone clone.
> + Also converts any nested git repositories (created by multiple SRC_URI git
> + entries with different destsuffix values) so that none of their contents
> + depend on the shared downloads directory via alternates.
> + """
> +
> import bb.process
> - if os.path.exists(os.path.join(repodir, '.git')):
> - alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
> +
> + def _convert(gitdir, workdir):
> + alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
> if os.path.exists(alternatesfile):
> # This will have been cloned with -s, so we need to convert it so none
> # of the contents is shared
> - bb.process.run('git repack -a', cwd=repodir)
> + bb.process.run('git repack -a', cwd=workdir)
> os.remove(alternatesfile)
>
> + if os.path.exists(os.path.join(repodir, '.git')):
> + _convert(os.path.join(repodir, '.git'), repodir)
> +
> + # Also handle nested git repos created by multiple SRC_URI git entries
> + # with different destsuffix values. Each nested repo is cloned with -s
> + # and has its own alternates pointing to the downloads directory.
> + for root, dirs, files in os.walk(repodir):
> + if root == repodir:
> + continue
> + if '.git' in dirs:
> + _convert(os.path.join(root, '.git'), root)
> + dirs[:] = [] # don't recurse into nested repos
Unfortunately, I don't think you can skip nested repos here. What about
a recipe which unpacks a git repository as a subdirectory of another?
This makes me think, instead of walking the directory tree under ${S}
looking for git repositories, could we walk the list of SRC_URI entries
looking for 'git://' URIs? That should be a much shorter list. If that
works, you'd probably want to do it in devtool-source.bbclass instead of
here.
Best regards,
--
Paul Barker
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe
2026-05-22 7:59 [PATCH v2 0/3] devtool: fix standalone clone conversion for nested git repos Jamin Lin
2026-05-22 7:59 ` [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace Jamin Lin
@ 2026-05-22 7:59 ` Jamin Lin
2026-05-22 10:44 ` Paul Barker
2026-05-22 7:59 ` [PATCH v2 3/3] oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos Jamin Lin
2 siblings, 1 reply; 6+ messages in thread
From: Jamin Lin @ 2026-05-22 7:59 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org, alex.kanavin@gmail.com,
paul@pbarker.dev
Cc: Troy Lee, Jamin Lin, Vince Chang
Add a test recipe with six git SRC_URI entries using nested destsuffix
values. This recipe is used by the devtool selftest
to verify that devtool modify correctly converts all nested git repos to
standalone clones.
Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
.../devtool-test-multi-destsuffix_git.bb | 29 +++++++++++++++++++
1 file changed, 29 insertions(+)
create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
new file mode 100644
index 0000000000..0702b13f3a
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
@@ -0,0 +1,29 @@
+SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values"
+LICENSE = "CLOSED"
+
+# Six git entries all nested inside S (${UNPACKDIR}). This exercises the
+# devtool code path that must convert every nested git repo to a standalone
+# clone: the initial fetch uses a shared clone whose alternates point into
+# downloads/git2/; git repack copies those objects locally so the workspace
+# survives 'bitbake -c cleanall'.
+SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \
+ git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \
+ git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \
+ git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \
+ git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \
+ git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \
+"
+
+SRCREV_level1_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level1_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level2_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level2_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level3_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level3_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_FORMAT = "level1_a_level1_b_level2_a_level2_b_level3_a_level3_b"
+
+S = "${UNPACKDIR}"
+
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe
2026-05-22 7:59 ` [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe Jamin Lin
@ 2026-05-22 10:44 ` Paul Barker
0 siblings, 0 replies; 6+ messages in thread
From: Paul Barker @ 2026-05-22 10:44 UTC (permalink / raw)
To: Jamin Lin, openembedded-core@lists.openembedded.org,
alex.kanavin@gmail.com
Cc: Troy Lee, Vince Chang
[-- Attachment #1: Type: text/plain, Size: 2426 bytes --]
On Fri, 2026-05-22 at 07:59 +0000, Jamin Lin wrote:
> Add a test recipe with six git SRC_URI entries using nested destsuffix
> values. This recipe is used by the devtool selftest
> to verify that devtool modify correctly converts all nested git repos to
> standalone clones.
>
> Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
> ---
> .../devtool-test-multi-destsuffix_git.bb | 29 +++++++++++++++++++
> 1 file changed, 29 insertions(+)
> create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
>
> diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
> new file mode 100644
> index 0000000000..0702b13f3a
> --- /dev/null
> +++ b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
> @@ -0,0 +1,29 @@
> +SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values"
> +LICENSE = "CLOSED"
> +
> +# Six git entries all nested inside S (${UNPACKDIR}). This exercises the
> +# devtool code path that must convert every nested git repo to a standalone
> +# clone: the initial fetch uses a shared clone whose alternates point into
> +# downloads/git2/; git repack copies those objects locally so the workspace
> +# survives 'bitbake -c cleanall'.
> +SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \
> + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \
> + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \
> + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \
> + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \
> + git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \
> +"
Thanks for including a test case and recipe!
These do not appear to be "nested" git repositories though, as none is
contained within another. We also don't need all 6 of them. Perhaps just
three is enough, nested like this:
- destsuffix=level1
- destsuffix=level1/level2
- destsuffix=level1/level2/level3
Best regards,
--
Paul Barker
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 3/3] oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos
2026-05-22 7:59 [PATCH v2 0/3] devtool: fix standalone clone conversion for nested git repos Jamin Lin
2026-05-22 7:59 ` [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace Jamin Lin
2026-05-22 7:59 ` [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe Jamin Lin
@ 2026-05-22 7:59 ` Jamin Lin
2 siblings, 0 replies; 6+ messages in thread
From: Jamin Lin @ 2026-05-22 7:59 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org, alex.kanavin@gmail.com,
paul@pbarker.dev
Cc: Troy Lee, Jamin Lin, Vince Chang
Add test_devtool_modify_multi_git_destsuffix_standalone to verify that
devtool modify converts all nested git repos (from multiple SRC_URI git
entries with different destsuffix values) to standalone clones so the
workspace survives 'bitbake -c cleanall'.
Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
meta/lib/oeqa/selftest/cases/devtool.py | 67 +++++++++++++++++++++++++
1 file changed, 67 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index 5ed69aee1b..b927bbf8f0 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -1182,6 +1182,73 @@ class DevtoolModifyTests(DevtoolBase):
self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake'), 'Submodule not found')
self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake-gitsm-test1'), 'Submodule not found')
+ def test_devtool_modify_multi_git_destsuffix_standalone(self):
+ """
+ Verify that devtool modify converts all nested git repos (from multiple
+ SRC_URI git entries with different destsuffix values) to standalone clones
+ so that 'bitbake -c cleanall' does not break the devtool workspace.
+
+ The recipe (devtool-test-multi-destsuffix) has six git SRC_URI entries
+ with S = ${UNPACKDIR}. All repos are nested inside S:
+ destsuffix=level1_a -> srcdir/level1_a/
+ destsuffix=level1_b -> srcdir/level1_b/
+ destsuffix=level1/level2_a -> srcdir/level1/level2_a/
+ destsuffix=level1/level2_b -> srcdir/level1/level2_b/
+ destsuffix=level1/level2/level3_a -> srcdir/level1/level2/level3_a/
+ destsuffix=level1/level2/level3_b -> srcdir/level1/level2/level3_b/
+
+ This mirrors real-world recipes that embed multiple module repos
+ as nested subdirectories of the primary source tree.
+ """
+ testrecipe = 'devtool-test-multi-destsuffix'
+ src_uri = get_bb_var('SRC_URI', testrecipe)
+ self.assertIn('git://', src_uri,
+ 'This test expects %s to have git SRC_URI entries' % testrecipe)
+ self.track_for_cleanup(self.workspacedir)
+ self.add_command_to_tearDown('devtool reset %s' % testrecipe)
+ self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+ result = runCmd('devtool modify %s' % testrecipe)
+ self.assertEqual(result.status, 0,
+ 'devtool modify failed: %s' % result.output)
+ srcdir = os.path.join(self.workspacedir, 'sources', testrecipe)
+ nested_paths = [
+ ('level1_a', 'level1_a'),
+ ('level1_b', 'level1_b'),
+ ('level2_a', 'level1/level2_a'),
+ ('level2_b', 'level1/level2_b'),
+ ('level3_a', 'level1/level2/level3_a'),
+ ('level3_b', 'level1/level2/level3_b'),
+ ]
+
+ for name, subpath in nested_paths:
+ repo_path = os.path.join(srcdir, subpath)
+ self.assertExists(os.path.join(repo_path, '.git'),
+ 'Repo %s (.git) not found in devtool workspace' % name)
+
+ # Key assertion: no nested repo should retain a git alternates file.
+ # devtool modify must repack objects locally so the workspace does not
+ # depend on the downloads cache, which 'bitbake -c cleanall' will delete.
+ for name, subpath in nested_paths:
+ repo_path = os.path.join(srcdir, subpath)
+ alternates_file = os.path.join(repo_path, '.git', 'objects',
+ 'info', 'alternates')
+ self.assertNotExists(alternates_file,
+ 'Repo %s still has a git alternates file after '
+ 'devtool modify' % name)
+
+ # Verify the workspace survives cleanall, which removes the shared
+ # objects in the downloads cache that alternates would reference.
+ bitbake('%s -c cleanall' % testrecipe)
+
+ # After cleanall all repos must still be usable.
+ # A broken alternates file would cause git operations to fail.
+ for name, subpath in nested_paths:
+ repo_path = os.path.join(srcdir, subpath)
+ result = runCmd('git status', cwd=repo_path)
+ self.assertEqual(result.status, 0,
+ 'git status failed in repo %s after cleanall: %s'
+ % (name, result.output))
+
class DevtoolUpdateTests(DevtoolBase):
def test_devtool_update_recipe(self):
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread