[PATCH v3 3/5] devtool-source: Convert nested git SRC_URI destsuffix repos to standalone clones

[Jamin Lin <jamin_lin@aspeedtech.com>]

When a recipe uses multiple git SRC_URI entries with different destsuffix
values (e.g. recipes with separate repositories for the kernel, modules
and application), do_unpack clones each source tree with
'git clone -n -s'.

The -s flag uses git's shared-object mechanism:
instead of copying objects locally it writes a .git/objects/info/alternates file
pointing back to the bare repository under the downloads directory (DL_DIR/git2/).

scriptutils.git_convert_standalone_clone() is called by devtool_post_unpack to
make the top-level source directory standalone: it runs 'git repack -a' to copy
all objects into the local object store and then removes the alternates file.

However it only processes the top-level source directory. Each nested git repo
created by a separate SRC_URI entry retains its own alternates file still
pointing into downloads/.

Steps to reproduce:
  1. devtool modify <recipe-with-multiple-git-SRC_URI>
  2. bitbake -c cleanall <recipe>
  3. bitbake <recipe>

At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes
the bare repositories from downloads/git2/. The top-level workspace
repo is standalone (alternates already removed by the original code),
but the nested repos still hold alternates pointing to the now-deleted
paths.

At step 3, srctree_hash_files() runs 'git add -A .' with a custom
GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on
each nested repo to check for changes; this fails with exit 128 because
the nested alternates are broken:
  error: unable to normalize alternate object path:
         .../downloads/git2/github.com.example.module//objects
  fatal: bad object HEAD
  fatal: 'git status --porcelain=2' failed in submodule modules/lib/module

This halts the BitBake parse phase with a CalledProcessError and leaves
the workspace in an unrecoverable state without manual intervention.

Fix by having devtool_post_unpack() look at the recipe's SRC_URI directly:
any git entry with an explicit destsuffix param names an additional
checkout nested under the source tree, so convert each of those to a
standalone clone the same way as the top-level tree.

This is deliberately metadata-driven rather than walking the unpacked
source tree looking for '.git' directories: a directory walk has no way
to tell a nested checkout from an ordinary subdirectory of one, so it
either has to stop at the first git repo it finds - which then misses a
destsuffix repo nested inside another repo's own working tree - or keep
walking into every repo's contents, which is wasted work for large trees.
Reading SRC_URI instead gives the exact, authoritative set of paths that
need converting, regardless of how they happen to be nested on disk.

Only entries with an explicit destsuffix are handled, since that is the
only way a recipe ends up with more than one git checkout under S; this
also avoids having to duplicate the git fetcher's internal logic for
computing an implicit default destsuffix (which depends on the subdir/
subpath params and BB_GIT_DEFAULT_DESTSUFFIX).

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 meta/classes/devtool-source.bbclass | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index f29f40588f..940cdedbac 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -97,6 +97,22 @@ python devtool_post_unpack() {
 
     scriptutils.git_convert_standalone_clone(srcsubdir)
 
+    # Recipes can use multiple git SRC_URI entries with an explicit destsuffix to
+    # unpack several repositories as nested subdirectories of the source tree
+    # (e.g. recipes with separate repos for the kernel, modules and
+    # application). Each such entry is unpacked as its own 'git clone -s' and
+    # needs the same standalone conversion as srcsubdir above, otherwise it keeps
+    # referencing objects in the downloads dir that 'bitbake -c cleanall' removes.
+    # We only look at entries with an explicit destsuffix param, since that's the
+    # only way a recipe ends up with more than one git checkout under S - this
+    # avoids having to duplicate the fetcher's internal default-destsuffix logic.
+    import bb.fetch2
+    fetch = bb.fetch2.Fetch(d.getVar('SRC_URI').split(), d)
+    for url in fetch.urls:
+        ud = fetch.ud[url]
+        if ud.type == 'git' and ud.parm.get('destsuffix'):
+            scriptutils.git_convert_standalone_clone(os.path.join(unpackdir, ud.parm['destsuffix']))
+
     # Make sure that srcsubdir exists
     bb.utils.mkdirhier(srcsubdir)
     if not os.listdir(srcsubdir):
-- 
2.43.0