[PATCH v2 0/3] devtool: fix standalone clone conversion for nested git repos

[PATCH v2 0/3] devtool: fix standalone clone conversion for nested git repos

[Jamin Lin]

When a recipe has multiple git SRC_URI entries with destsuffix values
nested inside S, devtool modify left all but the top-level repo with a
.git/objects/info/alternates file pointing into the downloads cache.

Running 'bitbake -c cleanall' then removed those shared objects, breaking
all subsequent git operations in the workspace.
Fix git_convert_standalone_clone() to walk all git repos nested inside S
and repack each one to a fully standalone clone.

v1:
  - Fix git_convert_standalone_clone() to walk all git repos nested
    inside S and repack each one to a standalone clone
v2:
  - Add selftest recipe devtool-test-multi-destsuffix with six nested
    git SRC_URI entries to reproduce the scenario
  - Add test_devtool_modify_multi_git_destsuffix_standalone to verify
    all nested repos have their alternates removed after devtool modify
    and remain usable after 'bitbake -c cleanall'

Jamin Lin (3):
  scripts/scriptutils: Convert nested git repos to standalone clones in
    devtool workspace
  meta-selftest: Add devtool-test-multi-destsuffix recipe
  oeqa/selftest/devtool: Add test for multiple nested git destsuffix
    repos

 .../devtool-test-multi-destsuffix_git.bb      | 29 ++++++++
 meta/lib/oeqa/selftest/cases/devtool.py       | 67 +++++++++++++++++++
 scripts/lib/scriptutils.py                    | 28 ++++++--
 3 files changed, 120 insertions(+), 4 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb

-- 
2.43.0

[PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

[Jamin Lin]

When a recipe uses multiple git SRC_URI entries with different destsuffix
values (e.g. Zephyr-based recipes with separate repositories for the
kernel, modules and application), do_unpack clones each source tree with
'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file
pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

git_convert_standalone_clone() is called by devtool_post_unpack to make
the workspace standalone: it runs 'git repack -a' to copy all objects
into the local object store and then removes the alternates file.

However it only processes the top-level source directory.  Each nested
git repo created by a separate SRC_URI entry retains its own alternates
file still pointing into downloads/.

Steps to reproduce:
  1. devtool modify <recipe-with-multiple-git-SRC_URI>
  2. bitbake -c cleanall <recipe>
  3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
the bare repositories from downloads/git2/. The top-level workspace
repo is standalone (alternates already removed by the original code),
but the nested repos still hold alternates pointing to the now-deleted
paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom
GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
each nested repo to check for changes; this fails with exit 128 because
the nested alternates are broken:
  error: unable to normalize alternate object path:
         .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
  fatal: bad object HEAD
  fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica

This halts the BitBake parse phase with a CalledProcessError and leaves
the workspace in an unrecoverable state without manual intervention.

Fix by extending git_convert_standalone_clone() to walk the source tree
and apply the same 'git repack -a' + remove-alternates treatment to
every nested git repository found, making the entire workspace fully
standalone at devtool modify time.  The walk uses dirs[:] = [] to stop
at each git boundary so it never descends into already-converted repos.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 32e749dbb1..0a83470373 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
 
 
 def git_convert_standalone_clone(repodir):
-    """If specified directory is a git repository, ensure it's a standalone clone"""
+    """
+    If specified directory is a git repository, ensure it's a standalone clone.
+    Also converts any nested git repositories (created by multiple SRC_URI git
+    entries with different destsuffix values) so that none of their contents
+    depend on the shared downloads directory via alternates.
+    """
+
     import bb.process
-    if os.path.exists(os.path.join(repodir, '.git')):
-        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
+
+    def _convert(gitdir, workdir):
+        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
         if os.path.exists(alternatesfile):
             # This will have been cloned with -s, so we need to convert it so none
             # of the contents is shared
-            bb.process.run('git repack -a', cwd=repodir)
+            bb.process.run('git repack -a', cwd=workdir)
             os.remove(alternatesfile)
 
+    if os.path.exists(os.path.join(repodir, '.git')):
+        _convert(os.path.join(repodir, '.git'), repodir)
+
+    # Also handle nested git repos created by multiple SRC_URI git entries
+    # with different destsuffix values. Each nested repo is cloned with -s
+    # and has its own alternates pointing to the downloads directory.
+    for root, dirs, files in os.walk(repodir):
+        if root == repodir:
+            continue
+        if '.git' in dirs:
+            _convert(os.path.join(root, '.git'), root)
+            dirs[:] = []  # don't recurse into nested repos
+
 def _get_temp_recipe_dir(d):
     # This is a little bit hacky but we need to find a place where we can put
     # the recipe so that bitbake can find it. We're going to delete it at the
-- 
2.43.0

[PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe

[Jamin Lin]

Add a test recipe with six git SRC_URI entries using nested destsuffix
values. This recipe is used by the devtool selftest
to verify that devtool modify correctly converts all nested git repos to
standalone clones.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 .../devtool-test-multi-destsuffix_git.bb      | 29 +++++++++++++++++++
 1 file changed, 29 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb

diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
new file mode 100644
index 0000000000..0702b13f3a
--- /dev/null
+++ b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
@@ -0,0 +1,29 @@
+SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values"
+LICENSE = "CLOSED"
+
+# Six git entries all nested inside S (${UNPACKDIR}). This exercises the
+# devtool code path that must convert every nested git repo to a standalone
+# clone: the initial fetch uses a shared clone whose alternates point into
+# downloads/git2/; git repack copies those objects locally so the workspace
+# survives 'bitbake -c cleanall'.
+SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \
+    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \
+"
+
+SRCREV_level1_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level1_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level2_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level2_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level3_a = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_level3_b = "64bc7c8fae61ded0c4e555aa775911f84c56e438"
+SRCREV_FORMAT = "level1_a_level1_b_level2_a_level2_b_level3_a_level3_b"
+
+S = "${UNPACKDIR}"
+
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
-- 
2.43.0

[PATCH v2 3/3] oeqa/selftest/devtool: Add test for multiple nested git destsuffix repos

[Jamin Lin]

Add test_devtool_modify_multi_git_destsuffix_standalone to verify that
devtool modify converts all nested git repos (from multiple SRC_URI git
entries with different destsuffix values) to standalone clones so the
workspace survives 'bitbake -c cleanall'.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py | 67 +++++++++++++++++++++++++
 1 file changed, 67 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index 5ed69aee1b..b927bbf8f0 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -1182,6 +1182,73 @@ class DevtoolModifyTests(DevtoolBase):
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake'), 'Submodule not found')
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake-gitsm-test1'), 'Submodule not found')
 
+    def test_devtool_modify_multi_git_destsuffix_standalone(self):
+        """
+        Verify that devtool modify converts all nested git repos (from multiple
+        SRC_URI git entries with different destsuffix values) to standalone clones
+        so that 'bitbake -c cleanall' does not break the devtool workspace.
+
+        The recipe (devtool-test-multi-destsuffix) has six git SRC_URI entries
+        with S = ${UNPACKDIR}. All repos are nested inside S:
+            destsuffix=level1_a              -> srcdir/level1_a/
+            destsuffix=level1_b              -> srcdir/level1_b/
+            destsuffix=level1/level2_a       -> srcdir/level1/level2_a/
+            destsuffix=level1/level2_b       -> srcdir/level1/level2_b/
+            destsuffix=level1/level2/level3_a -> srcdir/level1/level2/level3_a/
+            destsuffix=level1/level2/level3_b -> srcdir/level1/level2/level3_b/
+
+        This mirrors real-world recipes that embed multiple module repos
+        as nested subdirectories of the primary source tree.
+        """
+        testrecipe = 'devtool-test-multi-destsuffix'
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri,
+                      'This test expects %s to have git SRC_URI entries' % testrecipe)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('devtool reset %s' % testrecipe)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool modify %s' % testrecipe)
+        self.assertEqual(result.status, 0,
+                         'devtool modify failed: %s' % result.output)
+        srcdir = os.path.join(self.workspacedir, 'sources', testrecipe)
+        nested_paths = [
+            ('level1_a', 'level1_a'),
+            ('level1_b', 'level1_b'),
+            ('level2_a', 'level1/level2_a'),
+            ('level2_b', 'level1/level2_b'),
+            ('level3_a', 'level1/level2/level3_a'),
+            ('level3_b', 'level1/level2/level3_b'),
+        ]
+
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            self.assertExists(os.path.join(repo_path, '.git'),
+                              'Repo %s (.git) not found in devtool workspace' % name)
+
+        # Key assertion: no nested repo should retain a git alternates file.
+        # devtool modify must repack objects locally so the workspace does not
+        # depend on the downloads cache, which 'bitbake -c cleanall' will delete.
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            alternates_file = os.path.join(repo_path, '.git', 'objects',
+                                           'info', 'alternates')
+            self.assertNotExists(alternates_file,
+                                 'Repo %s still has a git alternates file after '
+                                 'devtool modify' % name)
+
+        # Verify the workspace survives cleanall, which removes the shared
+        # objects in the downloads cache that alternates would reference.
+        bitbake('%s -c cleanall' % testrecipe)
+
+        # After cleanall all repos must still be usable.
+        # A broken alternates file would cause git operations to fail.
+        for name, subpath in nested_paths:
+            repo_path = os.path.join(srcdir, subpath)
+            result = runCmd('git status', cwd=repo_path)
+            self.assertEqual(result.status, 0,
+                             'git status failed in repo %s after cleanall: %s'
+                             % (name, result.output))
+
 class DevtoolUpdateTests(DevtoolBase):
 
     def test_devtool_update_recipe(self):
-- 
2.43.0

Re: [PATCH v2 1/3] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 5088 bytes --]

On Fri, 2026-05-22 at 07:59 +0000, Jamin Lin wrote:
> When a recipe uses multiple git SRC_URI entries with different destsuffix
> values (e.g. Zephyr-based recipes with separate repositories for the
> kernel, modules and application), do_unpack clones each source tree with
> 'git clone -n -s'.
> 
> The -s flag uses git's shared-object mechanism:
> instead of copying objects locally it writes a .git/objects/info/alternates file
> pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
> 
> git_convert_standalone_clone() is called by devtool_post_unpack to make
> the workspace standalone: it runs 'git repack -a' to copy all objects
> into the local object store and then removes the alternates file.
> 
> However it only processes the top-level source directory.  Each nested
> git repo created by a separate SRC_URI entry retains its own alternates
> file still pointing into downloads/.
> 
> Steps to reproduce:
>   1. devtool modify <recipe-with-multiple-git-SRC_URI>
>   2. bitbake -c cleanall <recipe>
>   3. bitbake <recipe>
> 
> At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
> the bare repositories from downloads/git2/. The top-level workspace
> repo is standalone (alternates already removed by the original code),
> but the nested repos still hold alternates pointing to the now-deleted
> paths.
> 
> At step 3, srctree_hash_files() runs 'git add -A .' with a custom
> GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
> each nested repo to check for changes; this fails with exit 128 because
> the nested alternates are broken:
>   error: unable to normalize alternate object path:
>          .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
>   fatal: bad object HEAD
>   fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica
> 
> This halts the BitBake parse phase with a CalledProcessError and leaves
> the workspace in an unrecoverable state without manual intervention.
> 
> Fix by extending git_convert_standalone_clone() to walk the source tree
> and apply the same 'git repack -a' + remove-alternates treatment to
> every nested git repository found, making the entire workspace fully
> standalone at devtool modify time.  The walk uses dirs[:] = [] to stop
> at each git boundary so it never descends into already-converted repos.
> 
> Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
> ---
>  scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
>  1 file changed, 24 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
> index 32e749dbb1..0a83470373 100644
> --- a/scripts/lib/scriptutils.py
> +++ b/scripts/lib/scriptutils.py
> @@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
>  
>  
>  def git_convert_standalone_clone(repodir):
> -    """If specified directory is a git repository, ensure it's a standalone clone"""
> +    """
> +    If specified directory is a git repository, ensure it's a standalone clone.
> +    Also converts any nested git repositories (created by multiple SRC_URI git
> +    entries with different destsuffix values) so that none of their contents
> +    depend on the shared downloads directory via alternates.
> +    """
> +
>      import bb.process
> -    if os.path.exists(os.path.join(repodir, '.git')):
> -        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
> +
> +    def _convert(gitdir, workdir):
> +        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
>          if os.path.exists(alternatesfile):
>              # This will have been cloned with -s, so we need to convert it so none
>              # of the contents is shared
> -            bb.process.run('git repack -a', cwd=repodir)
> +            bb.process.run('git repack -a', cwd=workdir)
>              os.remove(alternatesfile)
>  
> +    if os.path.exists(os.path.join(repodir, '.git')):
> +        _convert(os.path.join(repodir, '.git'), repodir)
> +
> +    # Also handle nested git repos created by multiple SRC_URI git entries
> +    # with different destsuffix values. Each nested repo is cloned with -s
> +    # and has its own alternates pointing to the downloads directory.
> +    for root, dirs, files in os.walk(repodir):
> +        if root == repodir:
> +            continue
> +        if '.git' in dirs:
> +            _convert(os.path.join(root, '.git'), root)
> +            dirs[:] = []  # don't recurse into nested repos

Unfortunately, I don't think you can skip nested repos here. What about
a recipe which unpacks a git repository as a subdirectory of another?

This makes me think, instead of walking the directory tree under ${S}
looking for git repositories, could we walk the list of SRC_URI entries
looking for 'git://' URIs? That should be a much shorter list. If that
works, you'd probably want to do it in devtool-source.bbclass instead of
here.

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH v2 2/3] meta-selftest: Add devtool-test-multi-destsuffix recipe

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 2426 bytes --]

On Fri, 2026-05-22 at 07:59 +0000, Jamin Lin wrote:
> Add a test recipe with six git SRC_URI entries using nested destsuffix
> values. This recipe is used by the devtool selftest
> to verify that devtool modify correctly converts all nested git repos to
> standalone clones.
> 
> Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
> ---
>  .../devtool-test-multi-destsuffix_git.bb      | 29 +++++++++++++++++++
>  1 file changed, 29 insertions(+)
>  create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
> 
> diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
> new file mode 100644
> index 0000000000..0702b13f3a
> --- /dev/null
> +++ b/meta-selftest/recipes-test/devtool/devtool-test-multi-destsuffix_git.bb
> @@ -0,0 +1,29 @@
> +SUMMARY = "Test recipe for multiple git SRC_URI entries with nested destsuffix values"
> +LICENSE = "CLOSED"
> +
> +# Six git entries all nested inside S (${UNPACKDIR}). This exercises the
> +# devtool code path that must convert every nested git repo to a standalone
> +# clone: the initial fetch uses a shared clone whose alternates point into
> +# downloads/git2/; git repack copies those objects locally so the workspace
> +# survives 'bitbake -c cleanall'.
> +SRC_URI = "git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_a;destsuffix=level1_a \
> +    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level1_b;destsuffix=level1_b \
> +    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_a;destsuffix=level1/level2_a \
> +    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level2_b;destsuffix=level1/level2_b \
> +    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_a;destsuffix=level1/level2/level3_a \
> +    git://git.yoctoproject.org/dbus-wait;nobranch=1;protocol=https;name=level3_b;destsuffix=level1/level2/level3_b \
> +"

Thanks for including a test case and recipe!

These do not appear to be "nested" git repositories though, as none is
contained within another. We also don't need all 6 of them. Perhaps just
three is enough, nested like this:

  - destsuffix=level1
  - destsuffix=level1/level2
  - destsuffix=level1/level2/level3

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]