[PATCH v1] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

[PATCH v1] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

[Jamin Lin]

When a recipe uses multiple git SRC_URI entries with different destsuffix
values (e.g. Zephyr-based recipes with separate repositories for the
kernel, modules and application), do_unpack clones each source tree with
'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file
pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

git_convert_standalone_clone() is called by devtool_post_unpack to make
the workspace standalone: it runs 'git repack -a' to copy all objects
into the local object store and then removes the alternates file.

However it only processes the top-level source directory.  Each nested
git repo created by a separate SRC_URI entry retains its own alternates
file still pointing into downloads/.

Steps to reproduce:
  1. devtool modify <recipe-with-multiple-git-SRC_URI>
  2. bitbake -c cleanall <recipe>
  3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
the bare repositories from downloads/git2/. The top-level workspace
repo is standalone (alternates already removed by the original code),
but the nested repos still hold alternates pointing to the now-deleted
paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom
GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
each nested repo to check for changes; this fails with exit 128 because
the nested alternates are broken:
  error: unable to normalize alternate object path:
         .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
  fatal: bad object HEAD
  fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica

This halts the BitBake parse phase with a CalledProcessError and leaves
the workspace in an unrecoverable state without manual intervention.

Fix by extending git_convert_standalone_clone() to walk the source tree
and apply the same 'git repack -a' + remove-alternates treatment to
every nested git repository found, making the entire workspace fully
standalone at devtool modify time.  The walk uses dirs[:] = [] to stop
at each git boundary so it never descends into already-converted repos.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 32e749dbb1..0a83470373 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
 
 
 def git_convert_standalone_clone(repodir):
-    """If specified directory is a git repository, ensure it's a standalone clone"""
+    """
+    If specified directory is a git repository, ensure it's a standalone clone.
+    Also converts any nested git repositories (created by multiple SRC_URI git
+    entries with different destsuffix values) so that none of their contents
+    depend on the shared downloads directory via alternates.
+    """
+
     import bb.process
-    if os.path.exists(os.path.join(repodir, '.git')):
-        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
+
+    def _convert(gitdir, workdir):
+        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
         if os.path.exists(alternatesfile):
             # This will have been cloned with -s, so we need to convert it so none
             # of the contents is shared
-            bb.process.run('git repack -a', cwd=repodir)
+            bb.process.run('git repack -a', cwd=workdir)
             os.remove(alternatesfile)
 
+    if os.path.exists(os.path.join(repodir, '.git')):
+        _convert(os.path.join(repodir, '.git'), repodir)
+
+    # Also handle nested git repos created by multiple SRC_URI git entries
+    # with different destsuffix values. Each nested repo is cloned with -s
+    # and has its own alternates pointing to the downloads directory.
+    for root, dirs, files in os.walk(repodir):
+        if root == repodir:
+            continue
+        if '.git' in dirs:
+            _convert(os.path.join(root, '.git'), root)
+            dirs[:] = []  # don't recurse into nested repos
+
 def _get_temp_recipe_dir(d):
     # This is a little bit hacky but we need to find a place where we can put
     # the recipe so that bitbake can find it. We're going to delete it at the
-- 
2.43.0

Re: [OE-core] [PATCH v1] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

[Alexander Kanavin]

There have been patches fixing similar issues, for which oe-core has
no tests or ways to reproduce. Rather than ask reviewers to set up
zephyr builds so they can observe the issue, can you add a testcase
for this to devtool's oe-selftest?

Alex

On Wed, 20 May 2026 at 07:18, Jamin Lin via lists.openembedded.org
<jamin_lin=aspeedtech.com@lists.openembedded.org> wrote:
>
> When a recipe uses multiple git SRC_URI entries with different destsuffix
> values (e.g. Zephyr-based recipes with separate repositories for the
> kernel, modules and application), do_unpack clones each source tree with
> 'git clone -n -s'.
>
> The -s flag uses git's shared-object mechanism:
> instead of copying objects locally it writes a .git/objects/info/alternates file
> pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
>
> git_convert_standalone_clone() is called by devtool_post_unpack to make
> the workspace standalone: it runs 'git repack -a' to copy all objects
> into the local object store and then removes the alternates file.
>
> However it only processes the top-level source directory.  Each nested
> git repo created by a separate SRC_URI entry retains its own alternates
> file still pointing into downloads/.
>
> Steps to reproduce:
>   1. devtool modify <recipe-with-multiple-git-SRC_URI>
>   2. bitbake -c cleanall <recipe>
>   3. bitbake <recipe>
>
> At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
> the bare repositories from downloads/git2/. The top-level workspace
> repo is standalone (alternates already removed by the original code),
> but the nested repos still hold alternates pointing to the now-deleted
> paths.
>
> At step 3, srctree_hash_files() runs 'git add -A .' with a custom
> GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
> each nested repo to check for changes; this fails with exit 128 because
> the nested alternates are broken:
>   error: unable to normalize alternate object path:
>          .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
>   fatal: bad object HEAD
>   fatal: 'git status --porcelain=2' failed in submodule modules/lib/acpica
>
> This halts the BitBake parse phase with a CalledProcessError and leaves
> the workspace in an unrecoverable state without manual intervention.
>
> Fix by extending git_convert_standalone_clone() to walk the source tree
> and apply the same 'git repack -a' + remove-alternates treatment to
> every nested git repository found, making the entire workspace fully
> standalone at devtool modify time.  The walk uses dirs[:] = [] to stop
> at each git boundary so it never descends into already-converted repos.
>
> Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
> ---
>  scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
>  1 file changed, 24 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
> index 32e749dbb1..0a83470373 100644
> --- a/scripts/lib/scriptutils.py
> +++ b/scripts/lib/scriptutils.py
> @@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
>
>
>  def git_convert_standalone_clone(repodir):
> -    """If specified directory is a git repository, ensure it's a standalone clone"""
> +    """
> +    If specified directory is a git repository, ensure it's a standalone clone.
> +    Also converts any nested git repositories (created by multiple SRC_URI git
> +    entries with different destsuffix values) so that none of their contents
> +    depend on the shared downloads directory via alternates.
> +    """
> +
>      import bb.process
> -    if os.path.exists(os.path.join(repodir, '.git')):
> -        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info', 'alternates')
> +
> +    def _convert(gitdir, workdir):
> +        alternatesfile = os.path.join(gitdir, 'objects', 'info', 'alternates')
>          if os.path.exists(alternatesfile):
>              # This will have been cloned with -s, so we need to convert it so none
>              # of the contents is shared
> -            bb.process.run('git repack -a', cwd=repodir)
> +            bb.process.run('git repack -a', cwd=workdir)
>              os.remove(alternatesfile)
>
> +    if os.path.exists(os.path.join(repodir, '.git')):
> +        _convert(os.path.join(repodir, '.git'), repodir)
> +
> +    # Also handle nested git repos created by multiple SRC_URI git entries
> +    # with different destsuffix values. Each nested repo is cloned with -s
> +    # and has its own alternates pointing to the downloads directory.
> +    for root, dirs, files in os.walk(repodir):
> +        if root == repodir:
> +            continue
> +        if '.git' in dirs:
> +            _convert(os.path.join(root, '.git'), root)
> +            dirs[:] = []  # don't recurse into nested repos
> +
>  def _get_temp_recipe_dir(d):
>      # This is a little bit hacky but we need to find a place where we can put
>      # the recipe so that bitbake can find it. We're going to delete it at the
> --
> 2.43.0
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#237382): https://lists.openembedded.org/g/openembedded-core/message/237382
> Mute This Topic: https://lists.openembedded.org/mt/119403462/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

RE: [OE-core] [PATCH v1] scripts/scriptutils: Convert nested git repos to standalone clones in devtool workspace

[Jamin Lin]

Hi Alexander

Thanks for the review and suggestion.
Will add and resend v2.

Thanks,
Jamin

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: Wednesday, May 20, 2026 5:24 PM
> To: Jamin Lin <jamin_lin@aspeedtech.com>
> Cc: openembedded-core@lists.openembedded.org; Troy Lee
> <troy_lee@aspeedtech.com>; Vince Chang <vince_chang@aspeedtech.com>;
> paul@pbarker.dev
> Subject: Re: [OE-core] [PATCH v1] scripts/scriptutils: Convert nested git repos to
> standalone clones in devtool workspace
> 
> There have been patches fixing similar issues, for which oe-core has no tests or
> ways to reproduce. Rather than ask reviewers to set up zephyr builds so they
> can observe the issue, can you add a testcase for this to devtool's oe-selftest?
> 
> Alex
> 
> On Wed, 20 May 2026 at 07:18, Jamin Lin via lists.openembedded.org
> <jamin_lin=aspeedtech.com@lists.openembedded.org> wrote:
> >
> > When a recipe uses multiple git SRC_URI entries with different
> > destsuffix values (e.g. Zephyr-based recipes with separate
> > repositories for the kernel, modules and application), do_unpack
> > clones each source tree with 'git clone -n -s'.
> >
> > The -s flag uses git's shared-object mechanism:
> > instead of copying objects locally it writes a
> > .git/objects/info/alternates file pointing back to the bare repository under
> the downloads directory (DL_DIR/git2/).
> >
> > git_convert_standalone_clone() is called by devtool_post_unpack to
> > make the workspace standalone: it runs 'git repack -a' to copy all
> > objects into the local object store and then removes the alternates file.
> >
> > However it only processes the top-level source directory.  Each nested
> > git repo created by a separate SRC_URI entry retains its own
> > alternates file still pointing into downloads/.
> >
> > Steps to reproduce:
> >   1. devtool modify <recipe-with-multiple-git-SRC_URI>
> >   2. bitbake -c cleanall <recipe>
> >   3. bitbake <recipe>
> >
> > At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
> > the bare repositories from downloads/git2/. The top-level workspace
> > repo is standalone (alternates already removed by the original code),
> > but the nested repos still hold alternates pointing to the now-deleted
> > paths.
> >
> > At step 3, srctree_hash_files() runs 'git add -A .' with a custom
> > GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
> > each nested repo to check for changes; this fails with exit 128
> > because the nested alternates are broken:
> >   error: unable to normalize alternate object path:
> >          .../downloads/git2/github.com.zephyrproject-rtos.acpica//objects
> >   fatal: bad object HEAD
> >   fatal: 'git status --porcelain=2' failed in submodule
> > modules/lib/acpica
> >
> > This halts the BitBake parse phase with a CalledProcessError and
> > leaves the workspace in an unrecoverable state without manual intervention.
> >
> > Fix by extending git_convert_standalone_clone() to walk the source
> > tree and apply the same 'git repack -a' + remove-alternates treatment
> > to every nested git repository found, making the entire workspace
> > fully standalone at devtool modify time.  The walk uses dirs[:] = []
> > to stop at each git boundary so it never descends into already-converted
> repos.
> >
> > Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
> > ---
> >  scripts/lib/scriptutils.py | 28 ++++++++++++++++++++++++----
> >  1 file changed, 24 insertions(+), 4 deletions(-)
> >
> > diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
> > index 32e749dbb1..0a83470373 100644
> > --- a/scripts/lib/scriptutils.py
> > +++ b/scripts/lib/scriptutils.py
> > @@ -100,16 +100,36 @@ def load_plugins(logger, plugins, pluginpath):
> >
> >
> >  def git_convert_standalone_clone(repodir):
> > -    """If specified directory is a git repository, ensure it's a standalone
> clone"""
> > +    """
> > +    If specified directory is a git repository, ensure it's a standalone clone.
> > +    Also converts any nested git repositories (created by multiple SRC_URI
> git
> > +    entries with different destsuffix values) so that none of their contents
> > +    depend on the shared downloads directory via alternates.
> > +    """
> > +
> >      import bb.process
> > -    if os.path.exists(os.path.join(repodir, '.git')):
> > -        alternatesfile = os.path.join(repodir, '.git', 'objects', 'info',
> 'alternates')
> > +
> > +    def _convert(gitdir, workdir):
> > +        alternatesfile = os.path.join(gitdir, 'objects', 'info',
> > + 'alternates')
> >          if os.path.exists(alternatesfile):
> >              # This will have been cloned with -s, so we need to convert it
> so none
> >              # of the contents is shared
> > -            bb.process.run('git repack -a', cwd=repodir)
> > +            bb.process.run('git repack -a', cwd=workdir)
> >              os.remove(alternatesfile)
> >
> > +    if os.path.exists(os.path.join(repodir, '.git')):
> > +        _convert(os.path.join(repodir, '.git'), repodir)
> > +
> > +    # Also handle nested git repos created by multiple SRC_URI git entries
> > +    # with different destsuffix values. Each nested repo is cloned with -s
> > +    # and has its own alternates pointing to the downloads directory.
> > +    for root, dirs, files in os.walk(repodir):
> > +        if root == repodir:
> > +            continue
> > +        if '.git' in dirs:
> > +            _convert(os.path.join(root, '.git'), root)
> > +            dirs[:] = []  # don't recurse into nested repos
> > +
> >  def _get_temp_recipe_dir(d):
> >      # This is a little bit hacky but we need to find a place where we can
> put
> >      # the recipe so that bitbake can find it. We're going to delete
> > it at the
> > --
> > 2.43.0
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#237382):
> > https://lists.openembedded.org/g/openembedded-core/message/237382
> > Mute This Topic: https://lists.openembedded.org/mt/119403462/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> > [alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >