Re: [PATCH v4 3/5] devtool-source: Make nested destsuffix git repos standalone

[Paul Barker <paul@pbarker.dev>]

On Fri, 2026-07-31 at 09:26 +0000, Jamin Lin wrote:
> When a recipe uses multiple git SRC_URI entries with different destsuffix
> values (e.g. recipes with separate repositories for the kernel, modules
> and application), do_unpack clones each source tree with
> 'git clone -n -s'.
> 
> The -s flag uses git's shared-object mechanism:
> instead of copying objects locally it writes a .git/objects/info/alternates file
> pointing back to the bare repository under the downloads directory (DL_DIR/git2/).
> 
> scriptutils.git_convert_standalone_clone() is called by devtool_post_unpack to
> make the top-level source directory standalone: it runs 'git repack -a' to copy
> all objects into the local object store and then removes the alternates file.
> 
> However it only processes the top-level source directory. Each nested git repo
> created by a separate SRC_URI entry retains its own alternates file still
> pointing into downloads/.
> 
> Steps to reproduce:
>   1. devtool modify <recipe-with-multiple-git-SRC_URI>
>   2. bitbake -c cleanall <recipe>
>   3. bitbake <recipe>
> 
> At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
> the bare repositories from downloads/git2/. The top-level workspace
> repo is standalone (alternates already removed by the original code),
> but the nested repos still hold alternates pointing to the now-deleted
> paths.
> 
> At step 3, srctree_hash_files() runs 'git add -A .' with a custom
> GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
> each nested repo to check for changes; this fails with exit 128 because
> the nested alternates are broken:
>   error: unable to normalize alternate object path:
>          .../downloads/git2/github.com.example.module//objects
>   fatal: bad object HEAD
>   fatal: 'git status --porcelain=2' failed in submodule modules/lib/module
> 
> This halts the BitBake parse phase with a CalledProcessError and leaves
> the workspace in an unrecoverable state without manual intervention.
> 
> Fix by having devtool_post_unpack() look at the recipe's SRC_URI directly:
> any git entry with an explicit destsuffix param names an additional
> checkout nested under the source tree, so convert each of those to a
> standalone clone the same way as the top-level tree.
> 
> This is deliberately metadata-driven rather than walking the unpacked
> source tree looking for '.git' directories: a directory walk has no way
> to tell a nested checkout from an ordinary subdirectory of one, so it
> either has to stop at the first git repo it finds - which then misses a
> destsuffix repo nested inside another repo's own working tree - or keep
> walking into every repo's contents, which is wasted work for large trees.
> Reading SRC_URI instead gives the exact, authoritative set of paths that
> need converting, regardless of how they happen to be nested on disk.
> 
> Only entries with an explicit destsuffix are handled, since that is the
> only way a recipe ends up with more than one git checkout under S; this
> also avoids having to duplicate the git fetcher's internal logic for
> computing an implicit default destsuffix (which depends on the subdir/
> subpath params and BB_GIT_DEFAULT_DESTSUFFIX).
> 
> Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
> ---
>  meta/classes/devtool-source.bbclass | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
> index f29f40588f..940cdedbac 100644
> --- a/meta/classes/devtool-source.bbclass
> +++ b/meta/classes/devtool-source.bbclass
> @@ -97,6 +97,22 @@ python devtool_post_unpack() {
>  
>      scriptutils.git_convert_standalone_clone(srcsubdir)
>  
> +    # Recipes can use multiple git SRC_URI entries with an explicit destsuffix to
> +    # unpack several repositories as nested subdirectories of the source tree
> +    # (e.g. recipes with separate repos for the kernel, modules and
> +    # application). Each such entry is unpacked as its own 'git clone -s' and
> +    # needs the same standalone conversion as srcsubdir above, otherwise it keeps
> +    # referencing objects in the downloads dir that 'bitbake -c cleanall' removes.
> +    # We only look at entries with an explicit destsuffix param, since that's the
> +    # only way a recipe ends up with more than one git checkout under S - this
> +    # avoids having to duplicate the fetcher's internal default-destsuffix logic.
> +    import bb.fetch2
> +    fetch = bb.fetch2.Fetch(d.getVar('SRC_URI').split(), d)
> +    for url in fetch.urls:
> +        ud = fetch.ud[url]
> +        if ud.type == 'git' and ud.parm.get('destsuffix'):
> +            scriptutils.git_convert_standalone_clone(os.path.join(unpackdir, ud.parm['destsuffix']))
> +
>      # Make sure that srcsubdir exists
>      bb.utils.mkdirhier(srcsubdir)
>      if not os.listdir(srcsubdir):

Why still handle srcsubdir separately above this loop?

What about SRC_URI containing multiple git repositories that unpack
side-by-side instead of nested?

It may be better to just iterate through all git repositories in SRC_URI
rather than just the nested ones.

Best regards,

-- 
Paul Barker