[PATCH v1 0/2] oe: Fix build failures with multiple git SRC_URI entries

[PATCH v1 0/2] oe: Fix build failures with multiple git SRC_URI entries

[Jamin Lin]

Some recipes (e.g. Zephyr-based) use multiple git SRC_URI entries with
different destsuffix values, causing each source to be unpacked into a
separate subdirectory of EXTERNALSRC that retains its own .git directory.

These nested git repositories trigger two independent failures:

1. externalsrc.bbclass: 'git add -A .' exits with code 128 during
   srctree_hash_files(), halting the bitbake parse phase.

2. oe/reproducible.py: 'git log -1' exits with code 128 inside a nested
   repo found by find_git_repositories(), aborting do_unpack.

Jamin Lin (2):
  externalsrc: Handle nested git repos from multiple SRC_URI entries
  reproducible: Handle nested git repos in find_git_repositories

 meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++-
 meta/lib/oe/reproducible.py      |  6 +++++-
 2 files changed, 41 insertions(+), 2 deletions(-)

-- 
2.43.0

[PATCH v1 1/2] externalsrc: Handle nested git repos from multiple SRC_URI entries

[Jamin Lin]

When a recipe uses multiple git SRC_URI entries with different
destsuffix values (e.g. Zephyr-based recipes with separate repos for
the kernel, modules, and application), each source is unpacked into a
subdirectory of EXTERNALSRC that retains its own .git directory.

srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root,
which fails with exit code 128 when git encounters these unregistered
nested git repositories, halting the bitbake parse phase.

Fix by scanning for nested git repos before the add. If any are found,
exclude them from the top-level 'git add' using pathspec magic
':(exclude)<path>' and hash each nested repo independently using a
temporary index. This ensures changes in any nested repo still trigger
do_compile/do_configure to re-run.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 meta/classes/externalsrc.bbclass | 37 +++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 902ff2604f..0dd57af668 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -234,8 +234,43 @@ def srctree_hash_files(d, srcdir=None):
             # Update our custom index
             env = os.environ.copy()
             env['GIT_INDEX_FILE'] = tmp_index.name
-            subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
+            # Find nested git repos created by multiple SRC_URI git entries with
+            # different destsuffix values. git add -A . exits 128 when it encounters
+            # these unregistered nested repos.
+            nested_git_dirs = []
+            for root, dirs, files in os.walk(s_dir):
+                if root == s_dir:
+                    continue
+                if '.git' in dirs or '.git' in files:
+                    nested_git_dirs.append(root)
+                    dirs[:] = []  # don't recurse into nested repos
+            if nested_git_dirs:
+                excludes = [':(exclude)' + os.path.relpath(n, s_dir) for n in nested_git_dirs]
+                subprocess.check_output(['git', 'add', '-A', '.'] + excludes, cwd=s_dir, env=env)
+            else:
+                subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
             git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
+            # Hash each nested git repo separately so source changes there still
+            # trigger do_compile/do_configure to re-run.
+            for nested in nested_git_dirs:
+                nested_git = os.path.join(nested, '.git')
+                if not os.path.isdir(nested_git):
+                    continue
+                with tempfile.NamedTemporaryFile(prefix='oe-devtool-nested-index') as nested_tmp:
+                    nested_index = os.path.join(nested_git, 'index')
+                    if os.path.exists(nested_index):
+                        shutil.copyfile(nested_index, nested_tmp.name)
+                    nested_env = os.environ.copy()
+                    nested_env['GIT_INDEX_FILE'] = nested_tmp.name
+                    proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=nested,
+                                           env=nested_env, stdout=subprocess.DEVNULL,
+                                           stderr=subprocess.DEVNULL)
+                    proc.communicate()
+                    proc = subprocess.Popen(['git', 'write-tree'], cwd=nested,
+                                           env=nested_env, stdout=subprocess.PIPE,
+                                           stderr=subprocess.DEVNULL)
+                    stdout, _ = proc.communicate()
+                    git_sha1 += stdout.decode("utf-8")
             if os.path.exists(os.path.join(s_dir, ".gitmodules")) and os.path.getsize(os.path.join(s_dir, ".gitmodules")) > 0:
                 submodule_helper = subprocess.check_output(["git", "config", "--file", ".gitmodules", "--get-regexp", "path"], cwd=s_dir, env=env).decode("utf-8")
                 for line in submodule_helper.splitlines():
-- 
2.43.0

[PATCH v1 2/2] reproducible: Handle nested git repos in find_git_repositories

[Jamin Lin]

When EXTERNALSRC contains multiple nested git repositories (from
multiple SRC_URI git entries with different destsuffix values),
find_git_repositories() walks into sub-repos and
get_source_date_epoch_from_git() subsequently fails with exit code 128
when running 'git log -1' inside them.

Two fixes:
- Stop os.walk recursion when a .git entry is found (dirs[:] = []) to
  avoid descending into nested repos.
- Change 'git log -1' from check=True to check=False with explicit
  error handling, so a failing nested repo is skipped gracefully
  instead of raising CalledProcessError and aborting do_unpack.

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
---
 meta/lib/oe/reproducible.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/reproducible.py b/meta/lib/oe/reproducible.py
index a80376010a..6bb25da55a 100644
--- a/meta/lib/oe/reproducible.py
+++ b/meta/lib/oe/reproducible.py
@@ -82,6 +82,7 @@ def find_git_repositories(d, sourcedir):
         for root, dirs, files in os.walk(mainpath, topdown=True):
             if '.git' in dirs or '.git' in files:
                 git_repositories.append(root)
+                dirs[:] = []  # don't recurse into nested git repos (multiple SRC_URI destsuffix)
 
     if not git_repositories:
         bb.warn('Failed to find any git repositories in UNPACKDIR or S')
@@ -105,7 +106,10 @@ def get_source_date_epoch_from_git(d, sourcedir):
 
         bb.debug(1, "git repository: %s" % repo_path)
         p = subprocess.run(['git', '-C', repo_path, 'log', '-1', '--pretty=%ct'],
-                           check=True, stdout=subprocess.PIPE)
+                           check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+        if p.returncode != 0:
+            bb.debug(1, "git log failed for %s (exit %d): %s" % (repo_path, p.returncode, p.stdout.decode('utf-8')))
+            continue
         source_dates.append(int(p.stdout.decode('utf-8')))
 
     if source_dates:
-- 
2.43.0