[PATCH 0/4] bitbake-layers: layerindex-fetch: respect --branch for already-configured layers

[PATCH 0/4] bitbake-layers: layerindex-fetch: respect --branch for already-configured layers

[Osama Abdelkader]

[YOCTO #7852]

This patch series fixes issues with the layerindex-fetch command when --branch
is specified for layers that are already configured in bblayers.conf.

The series is organized with bug fixes first (for potential backporting),
followed by feature additions:

1. Fix undefined stderr variable (bug fix - may need backporting)
2. Fix branch detection method (bug fix)
3. Respect --branch for already-configured layers (feature)
4. Add branch check/switch for cooker layers (feature)

The main issues addressed:
- Early exit without branch verification when layer is already configured
- Cooker layers being skipped entirely, preventing branch checking
- Fragile branch detection using 'git branch' output parsing
- Undefined variable bug in error handling

Osama Abdelkader (4):
  bitbake-layers: layerindex-fetch: Fix undefined stderr variable
  bitbake-layers: layerindex-fetch: Fix branch detection method
  bitbake-layers: layerindex-fetch: Respect --branch for
    already-configured layers
  bitbake-layers: layerindex-fetch: Add branch check/switch for cooker
    layers

 bitbake/lib/bblayers/layerindex.py | 51 ++++++++++++++++++++++++++----
 1 file changed, 45 insertions(+), 6 deletions(-)

-- 
2.43.0

[PATCH 1/4] bitbake-layers: layerindex-fetch: Fix undefined stderr variable

[Osama Abdelkader]

Fix undefined 'stderr' variable in error handling. The variable should
be 'completed_proc.stderr' to properly display the error message from the git command.

Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>
---
 lib/bblayers/layerindex.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
index ba91fac669..0ffc95822b 100644
--- a/lib/bblayers/layerindex.py
+++ b/lib/bblayers/layerindex.py
@@ -58,7 +58,7 @@ class LayerIndexPlugin(ActionPlugin):
             cmd = base_cmd + ['branch']
             completed_proc = subprocess.run(cmd, text=True, capture_output=True)
             if completed_proc.returncode:
-                logger.error("Unable to validate repo %s (%s)" % (repodir, stderr))
+                logger.error("Unable to validate repo %s (%s)" % (repodir, completed_proc.stderr))
                 return None, None, None
             else:
                 if branch != completed_proc.stdout[2:-1]:
-- 
2.43.0

[PATCH 2/4] bitbake-layers: layerindex-fetch: Fix branch detection method

[Osama Abdelkader]

Replace fragile parsing of 'git branch' output with the more reliable
'git rev-parse --abbrev-ref HEAD' command to get the current branch name.
This avoids parsing issues and is the recommended way to get the current branch name.

Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>
---
 lib/bblayers/layerindex.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
index 0ffc95822b..1c6511889d 100644
--- a/lib/bblayers/layerindex.py
+++ b/lib/bblayers/layerindex.py
@@ -55,13 +55,15 @@ class LayerIndexPlugin(ActionPlugin):
             switching branches if necessary and possible.
             """
             base_cmd = ['git', '--git-dir=%s/.git' % repodir, '--work-tree=%s' % repodir]
-            cmd = base_cmd + ['branch']
+            # Get current branch name
+            cmd = base_cmd + ['rev-parse', '--abbrev-ref', 'HEAD']
             completed_proc = subprocess.run(cmd, text=True, capture_output=True)
             if completed_proc.returncode:
                 logger.error("Unable to validate repo %s (%s)" % (repodir, completed_proc.stderr))
                 return None, None, None
             else:
-                if branch != completed_proc.stdout[2:-1]:
+                current_branch = completed_proc.stdout.strip()
+                if branch != current_branch:
                     cmd = base_cmd + ['status', '--short']
                     completed_proc = subprocess.run(cmd, text=True, capture_output=True)
                     if completed_proc.stdout.count('\n') != 0:
-- 
2.43.0

[PATCH 3/4] bitbake-layers: layerindex-fetch: Respect --branch for already-configured layers

[Osama Abdelkader]

When --branch is specified, prevent the early exit that skips branch
verification. This ensures that even if a layer is already in bblayers.conf,
the command will still check and switch to the requested branch if needed.

This fixes the issue where the command would exit early with "You already
have the requested layer(s)" without verifying the branch.

[YOCTO #7852]

Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>
---
 lib/bblayers/layerindex.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
index 1c6511889d..e0df630a7f 100644
--- a/lib/bblayers/layerindex.py
+++ b/lib/bblayers/layerindex.py
@@ -120,7 +120,8 @@ class LayerIndexPlugin(ActionPlugin):
 
         # Fast path, check if we already have what has been requested!
         (dependencies, invalidnames) = cookerIndex.find_dependencies(names=args.layername, ignores=ignore_layers)
-        if not args.show_only and not invalidnames:
+        if not args.show_only and not invalidnames and not args.branch:
+            # Only skip if no specific branch was requested
             logger.plain("You already have the requested layer(s): %s" % args.layername)
             return 0
 
-- 
2.43.0

[PATCH 4/4] bitbake-layers: layerindex-fetch: Add branch check/switch for cooker layers

[Osama Abdelkader]

For layers already in bblayers.conf (cooker layers), add logic to check
and switch branches when --branch is specified. Previously, cooker layers
were skipped entirely in the fetch loop, so branch checking never executed.

The code finds each layer's git repository root, determines the subdirectory
relative to the repo root, and calls get_fetch_layer() with the requested
branch to perform validation and checkout if needed.

This completes the fix started in previous commit, allowing users to
use --branch to switch already-configured layers to different branches.

[YOCTO #7852]

Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>
---
 lib/bblayers/layerindex.py | 40 ++++++++++++++++++++++++++++--
 1 file changed, 38 insertions(+), 2 deletions(-)

diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
index e0df630a7f..72d2eb8fb7 100644
--- a/lib/bblayers/layerindex.py
+++ b/lib/bblayers/layerindex.py
@@ -205,8 +205,44 @@ class LayerIndexPlugin(ActionPlugin):
             for deplayerbranch in dependencies:
                 layerBranch = dependencies[deplayerbranch][0]
 
-                if layerBranch.index.config['TYPE'] == 'cooker':
-                    # Anything loaded via cooker is already local, skip it
+                is_cooker_layer = layerBranch.index.config['TYPE'] == 'cooker'
+
+                if is_cooker_layer:
+                    # Anything loaded via cooker is already local
+                    # If a branch was specified, we still need to check/switch branches
+                    if args.branch:
+                        # Map collection names to layer paths (same as cooker plugin does)
+                        layerconfs = self.tinfoil.config_data.varhistory.get_variable_items_files('BBFILE_COLLECTIONS')
+                        bbfile_collections = {layer: os.path.dirname(os.path.dirname(path)) for layer, path in layerconfs.items()}
+
+                        # Get the layer path for this cooker layer's collection
+                        collection_name = layerBranch.collection
+                        if collection_name in bbfile_collections:
+                            bblayer = bbfile_collections[collection_name]
+                            if os.path.exists(bblayer):
+                                try:
+                                    # Find the git repository root
+                                    gitdir_cmd = ['git', '-C', bblayer, 'rev-parse', '--show-toplevel']
+                                    result = subprocess.run(gitdir_cmd, capture_output=True, text=True)
+                                    if result.returncode == 0:
+                                        repo_root = result.stdout.strip()
+                                        # Get the subdir relative to repo root
+                                        rel_subdir = os.path.relpath(bblayer, repo_root) if bblayer != repo_root else ''
+                                        # Get parent dir of repo as fetchdir
+                                        parent_fetchdir = os.path.dirname(repo_root)
+                                        # Now call get_fetch_layer to check/switch branch
+                                        # Use args.branch (the requested branch) not layerBranch.actual_branch (current branch)
+                                        _, name, _ = self.get_fetch_layer(parent_fetchdir,
+                                                                          layerBranch.layer.vcs_url,
+                                                                          rel_subdir,
+                                                                          not args.show_only,
+                                                                          args.branch,
+                                                                          args.shallow)
+                                        if not name:
+                                            # Error during branch check/switch
+                                            return 1
+                                except subprocess.SubprocessError:
+                                    pass  # Not a git repo, skip
                     continue
 
                 subdir, name, layerdir = self.get_fetch_layer(fetchdir,
-- 
2.43.0

Re: [PATCH 1/4] bitbake-layers: layerindex-fetch: Fix undefined stderr variable

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On Fri, 2025-12-12 at 20:08 +0100, Osama Abdelkader wrote:
> Fix undefined 'stderr' variable in error handling. The variable should
> be 'completed_proc.stderr' to properly display the error message from the git command.
> 
> Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>
> ---
>  lib/bblayers/layerindex.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
> index ba91fac669..0ffc95822b 100644
> --- a/lib/bblayers/layerindex.py
> +++ b/lib/bblayers/layerindex.py
> @@ -58,7 +58,7 @@ class LayerIndexPlugin(ActionPlugin):
>              cmd = base_cmd + ['branch']
>              completed_proc = subprocess.run(cmd, text=True, capture_output=True)
>              if completed_proc.returncode:
> -                logger.error("Unable to validate repo %s (%s)" % (repodir, stderr))
> +                logger.error("Unable to validate repo %s (%s)" % (repodir, completed_proc.stderr))
>                  return None, None, None
>              else:
>                  if branch != completed_proc.stdout[2:-1]:

Reviewed-by: Paul Barker <paul@pbarker.dev>

This should be applied for the bitbake stable branches as well as
master.

Best regards,

-- 
Paul Barker



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/4] bitbake-layers: layerindex-fetch: Fix branch detection method

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

On Fri, 2025-12-12 at 20:08 +0100, Osama Abdelkader wrote:
> Replace fragile parsing of 'git branch' output with the more reliable
> 'git rev-parse --abbrev-ref HEAD' command to get the current branch name.
> This avoids parsing issues and is the recommended way to get the current branch name.

Do you have a source for this recommendation? I thought the preferred
command was `git branch --show-current` as that just prints the branch
name with no extra formatting.

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/4] bitbake-layers: layerindex-fetch: Fix branch detection method

[Osama Abdelkader]

On Mon, Dec 15, 2025 at 11:54:20AM +0000, Paul Barker wrote:
> On Fri, 2025-12-12 at 20:08 +0100, Osama Abdelkader wrote:
> > Replace fragile parsing of 'git branch' output with the more reliable
> > 'git rev-parse --abbrev-ref HEAD' command to get the current branch name.
> > This avoids parsing issues and is the recommended way to get the current branch name.
> 
> Do you have a source for this recommendation? I thought the preferred
> command was `git branch --show-current` as that just prints the branch
> name with no extra formatting.

Git commands are divided into two groups:

Porcelain - High level commands intended to be used at the command line
Plumbing - Low level commands intended for use in scripts

git rev-parse is a "plumbing" command, meaning its output format is guaranteed to be stable across different versions of Git.
It reliably outputs only the raw branch name to standard output.

git branch is a "porcelain" command, intended for human users. While git branch --show-current currently behaves helpfully,
the formatting of porcelain commands is theoretically subject to change in future Git releases, which could break your build
scripts unexpectedly.

https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain
https://git-scm.com/docs/git-rev-parse

> 
> Best regards,
> 
> -- 
> Paul Barker
> 

Best regards,
Osama

Re: [PATCH 2/4] bitbake-layers: layerindex-fetch: Fix branch detection method

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]

On Mon, 2025-12-15 at 23:34 +0100, Osama Abdelkader wrote:
> On Mon, Dec 15, 2025 at 11:54:20AM +0000, Paul Barker wrote:
> > On Fri, 2025-12-12 at 20:08 +0100, Osama Abdelkader wrote:
> > > Replace fragile parsing of 'git branch' output with the more reliable
> > > 'git rev-parse --abbrev-ref HEAD' command to get the current branch name.
> > > This avoids parsing issues and is the recommended way to get the current branch name.
> > 
> > Do you have a source for this recommendation? I thought the preferred
> > command was `git branch --show-current` as that just prints the branch
> > name with no extra formatting.
> 
> Git commands are divided into two groups:
> 
> Porcelain - High level commands intended to be used at the command line
> Plumbing - Low level commands intended for use in scripts
> 
> git rev-parse is a "plumbing" command, meaning its output format is guaranteed to be stable across different versions of Git.
> It reliably outputs only the raw branch name to standard output.
> 
> git branch is a "porcelain" command, intended for human users. While git branch --show-current currently behaves helpfully,
> the formatting of porcelain commands is theoretically subject to change in future Git releases, which could break your build
> scripts unexpectedly.
> 
> https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain
> https://git-scm.com/docs/git-rev-parse

Hi Osama,

Thanks for the additional context! We discussed this on the patch review
call today.

`git rev-parse --abbrev-ref` is definitely an improvement over calling
`git branch`, but we have still seen some issues with this command in
other contexts on the autobuilder. The best command to use seems to be
`git name-rev --name-only` [1].

[1]: https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=1e80c19f62169c536e44f99fdcdf9f1e84c65d99

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/4] bitbake-layers: layerindex-fetch: Fix branch detection method

[Osama Abdelkader]

On Thu, Dec 18, 2025 at 07:39:28PM +0000, Paul Barker wrote:
> On Mon, 2025-12-15 at 23:34 +0100, Osama Abdelkader wrote:
> > On Mon, Dec 15, 2025 at 11:54:20AM +0000, Paul Barker wrote:
> > > On Fri, 2025-12-12 at 20:08 +0100, Osama Abdelkader wrote:
> > > > Replace fragile parsing of 'git branch' output with the more reliable
> > > > 'git rev-parse --abbrev-ref HEAD' command to get the current branch name.
> > > > This avoids parsing issues and is the recommended way to get the current branch name.
> > > 
> > > Do you have a source for this recommendation? I thought the preferred
> > > command was `git branch --show-current` as that just prints the branch
> > > name with no extra formatting.
> > 
> > Git commands are divided into two groups:
> > 
> > Porcelain - High level commands intended to be used at the command line
> > Plumbing - Low level commands intended for use in scripts
> > 
> > git rev-parse is a "plumbing" command, meaning its output format is guaranteed to be stable across different versions of Git.
> > It reliably outputs only the raw branch name to standard output.
> > 
> > git branch is a "porcelain" command, intended for human users. While git branch --show-current currently behaves helpfully,
> > the formatting of porcelain commands is theoretically subject to change in future Git releases, which could break your build
> > scripts unexpectedly.
> > 
> > https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain
> > https://git-scm.com/docs/git-rev-parse
> 
> Hi Osama,
> 
> Thanks for the additional context! We discussed this on the patch review
> call today.
> 
> `git rev-parse --abbrev-ref` is definitely an improvement over calling
> `git branch`, but we have still seen some issues with this command in
> other contexts on the autobuilder. The best command to use seems to be
> `git name-rev --name-only` [1].
> 
> [1]: https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=1e80c19f62169c536e44f99fdcdf9f1e84c65d99
> 
> Best regards,
> 
> -- 
> Paul Barker
> 

Hi Paul,

Thanks for the review, okay will change to that.

Best regards,
Osama