[PATCH v4 2/5] devtool: Register nested git repos before the initial commit

[Jamin Lin <jamin_lin@aspeedtech.com>]

setup_git_repo() is meant to convert a git repo that a recipe unpacks
inside S (e.g. via multiple git SRC_URI entries with different
destsuffix values) into a regular git submodule, so devtool can later
tag branches on it and extract patches from it via finish/update.

That detection never actually triggered, because of the order the
function ran things in when it had to create the workspace repo itself:

  1. 'git init'
  2. 'git add -A .' + initial commit  <- commits the nested repo as a
                                         bare, unregistered gitlink
  3. checkout devbranch, tag basetag
  4. scan 'git status --porcelain' for still-untracked directories
     ("?? <dir>/") and convert any that are git repos into submodules

By the time step 4 ran, the nested repo had already been swept up by
step 2's 'git add -A .': git treats a directory containing its own .git
as an embedded repo and stages it as a gitlink pointing at its current
HEAD, without registering it as a submodule. Once that gitlink is
committed, 'git status --porcelain' reports it as e.g. " M <dir>"
(already tracked) rather than "?? <dir>/" (untracked), so step 4's
"line.endswith('/')" check could never match it and the conversion to a
real submodule silently never happened.

There is also a second entry path with the same root cause: when the
recipe's top-level source is itself fetched via git://, repodir is
already a git repo, so the 'if not .git' block above (init + initial
commit) is skipped entirely - and so was the detection that lived inside
it. In that case the nested repo instead gets committed as a bare
gitlink later, by patch_task_postfunc's 'git add' after do_patch.

Fix this by extracting the detection into a helper and calling it before
anything can commit the nested repo as a bare gitlink, in both cases:
  - freshly-created workspace repo: right after 'git init', before
    'git add -A .' and the initial commit;
  - repodir already a git repo: at function entry, before the later
    'git add' in patch_task_postfunc.
At those points the nested repo is still untracked and reported with a
trailing "/", so it is correctly picked up and registered via
'git submodule add'.

Nested repos are discovered top-down (so a repo that manages its own
submodules via .gitmodules can be skipped rather than descended into),
but registered bottom-up (deepest first): a parent's commit recording
its child's HEAD must happen after that child is fully finalized,
otherwise registering a still-deeper repo afterwards moves the child's
HEAD forward again and leaves the parent pointing at a stale revision.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 scripts/lib/devtool/__init__.py | 79 +++++++++++++++++++++++----------
 1 file changed, 56 insertions(+), 23 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 58b02eb460..030ba7edd9 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -197,9 +197,59 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     """
     import bb.process
     import oe.patch
+
+    def register_nested_git_submodules():
+        # If the recipe unpacks another git repo inside S (e.g. multiple git
+        # SRC_URI entries with destsuffix), declare it as a regular git
+        # submodule now, so we will be able to tag branches on it and extract
+        # patches when doing finish/update on the recipe. This must happen
+        # before anything else commits that nested repo as a bare,
+        # unregistered gitlink: once that happens 'git status' no longer
+        # reports it as untracked ("?? <dir>/"), so this detection can never
+        # find it. That can happen either from 'git add -A .' below (for a
+        # freshly-initialized repo) or, when repodir is already its own git
+        # repo (e.g. a recipe fetched via plain git://), from a later 'git
+        # add' done elsewhere (patch_task_postfunc, after do_patch) - so this
+        # is called both from the fresh-repo branch below and from the
+        # already-a-repo branch, before either has a chance to do that.
+        #
+        # Discover nested repos top-down (so we can still skip descending into
+        # a repo that manages its own submodules via .gitmodules), but do the
+        # actual 'git submodule add' + commit bottom-up (deepest repo first):
+        # a parent's commit recording its child's current HEAD must happen
+        # after that child is fully finalized, otherwise a deeper repo added
+        # later on gets its own registration commit, moving the child's HEAD
+        # forward again and leaving the parent's already-made commit pointing
+        # at a stale, superseded revision of it.
+        stdout, _ = bb.process.run("git status --porcelain", cwd=repodir)
+        nested_repos = []
+        for line in stdout.splitlines():
+            if line.endswith("/"):
+                new_dir = line.split()[1]
+                for root, dirs, files in os.walk(os.path.join(repodir, new_dir)):
+                    if ".git" in dirs + files:
+                        nested_repos.append(root)
+                        # Do not descend into nested git repos that have submodules themselves.
+                        if ".gitmodules" in files:
+                            logger.warning('Nested git repository with submodules %s; devtool will not recurse into it', root)
+                            dirs[:] = []
+
+        for root in reversed(nested_repos):
+            parentdir = os.path.join(root, "..")
+            (stdout, _) = bb.process.run('git remote', cwd=root)
+            remote = stdout.splitlines()[0]
+            (stdout, _) = bb.process.run('git remote get-url %s' % remote, cwd=root)
+            remote_url = stdout.splitlines()[0]
+            logger.error(os.path.relpath(parentdir, root))
+            bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, parentdir)), cwd=parentdir)
+            oe.patch.GitApplyTree.commitIgnored("Add additional submodule from SRC_URI", dir=parentdir, d=d)
+
     if not os.path.exists(os.path.join(repodir, '.git')):
         bb.process.run('git init', cwd=repodir)
         bb.process.run('git config --local gc.autodetach 0', cwd=repodir)
+
+        register_nested_git_submodules()
+
         bb.process.run('git add -f -A .', cwd=repodir)
         commit_cmd = ['git']
         oe.patch.GitApplyTree.gitCommandUserOptions(commit_cmd, d=d)
@@ -214,6 +264,12 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
             commitmsg = "Initial commit from upstream"
         commit_cmd += ['-m', commitmsg]
         bb.process.run(commit_cmd, cwd=repodir)
+    else:
+        # repodir is already a git repo in its own right (e.g. a recipe whose
+        # top-level source is fetched via plain git://), so there was no
+        # fresh init/initial commit above to interfere with detecting nested
+        # repos - do it here instead, at the earliest point available.
+        register_nested_git_submodules()
 
     # Ensure singletask.lock (as used by externalsrc.bbclass) is ignored by git
     gitinfodir = os.path.join(repodir, '.git', 'info')
@@ -237,29 +293,6 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
     bb.process.run('git tag -f --no-sign %s' % basetag, cwd=repodir)
 
-    # if recipe unpacks another git repo inside S, we need to declare it as a regular git submodule now,
-    # so we will be able to tag branches on it and extract patches when doing finish/update on the recipe
-    stdout, _ = bb.process.run("git status --porcelain", cwd=repodir)
-    found = False
-    for line in stdout.splitlines():
-        if line.endswith("/"):
-            new_dir = line.split()[1]
-            for root, dirs, files in os.walk(os.path.join(repodir, new_dir)):
-                if ".git" in dirs + files:
-                    (stdout, _) = bb.process.run('git remote', cwd=root)
-                    remote = stdout.splitlines()[0]
-                    (stdout, _) = bb.process.run('git remote get-url %s' % remote, cwd=root)
-                    remote_url = stdout.splitlines()[0]
-                    logger.error(os.path.relpath(os.path.join(root, ".."), root))
-                    bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, os.path.join(root, ".."))), cwd=os.path.join(root, ".."))
-                    # Do not descend into nested git repos that have submodules themselves.
-                    if ".gitmodules" in files:
-                        logger.warning('Nested git repository with submodules %s; devtool will not recurse into it', root)
-                        dirs[:] = []
-                    found = True
-                if found:
-                    oe.patch.GitApplyTree.commitIgnored("Add additional submodule from SRC_URI", dir=os.path.join(root, ".."), d=d)
-                    found = False
     if os.path.exists(os.path.join(repodir, '.gitmodules')):
         bb.process.run('git submodule foreach --recursive  "git tag -f --no-sign %s"' % basetag, cwd=repodir)
 
-- 
2.43.0