[PATCH] fetch/git: add reference option

[PATCH] fetch/git: add reference option

[Daniel Walker]

Adds a option to the git fetcher which can be used to specify a local
reference for a given SRC_URI. Which is then fed into the clone command
as "--reference-if-able" which make the option fault tolerant if the
directory doesn't exist.

AI-Generated: OpenAI Codex
Signed-off-by: Daniel Walker <danielwa@cisco.com>
Change-Id: Ib5eb685ddfb48fa19f1c65cfc468894806f99e83
---
 .../bitbake-user-manual-fetching.rst                 |  7 +++++++
 lib/bb/fetch2/git.py                                 | 12 +++++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index b96018a4e..6e4fc0800 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -398,6 +398,13 @@ This fetcher supports the following parameters:
    preserve the local cache carefully for future use. The default value
    for this parameter is "0".
 
+-  *"reference":* Specifies a local directory to use as a reference
+   repository. When set, the fetcher passes the directory to ``git clone``
+   with ``--reference-if-able``. Git uses the reference repository's
+   objects when possible, reducing the amount of object data copied or
+   fetched. If the reference directory cannot be used, Git continues
+   cloning without it.
+
 -  *"nobranch":* Tells the fetcher to not check the SHA validation for
    the branch when set to "1". The default is "0". Set this option for
    the recipe that refers to the commit that is valid for any namespace
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 0fbd85441..df1c4415f 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -22,6 +22,10 @@ Supported SRC_URI options are:
    remind fetcher to preserve local cache carefully for future use.
    The default value is "0", set rebaseable=1 for rebaseable git repo.
 
+- reference
+   Use the specified local directory as a reference repository by passing it
+   to git clone with --reference-if-able.
+
 - nocheckout
    Don't checkout source code when unpacking. set this option for the recipe
    who has its own routine to checkout code.
@@ -195,6 +199,12 @@ class Git(FetchMethod):
         if ud.bareclone:
             ud.cloneflags.append("--mirror")
 
+        ud.reference_args = []
+        ud.reference = ud.parm.get("reference")
+        if ud.reference:
+            ud.reference_args = ["--reference-if-able", ud.reference]
+            ud.cloneflags.extend(ud.reference_args)
+
         ud.shallow_skip_fast = False
         ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1"
         ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split()
@@ -436,7 +446,7 @@ class Git(FetchMethod):
                 objects = os.path.join(repourl_path, 'objects')
                 if os.path.isdir(objects) and not os.path.islink(objects):
                     repourl = repourl_path
-            clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror', repourl, ud.clonedir, '--progress']
+            clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror'] + ud.reference_args + [repourl, ud.clonedir, '--progress']
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
-- 
2.44.0

Re: [bitbake-devel] [PATCH] fetch/git: add reference option

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 4986 bytes --]

On Fri, Jun 26, 2026 at 3:10 PM Daniel Walker via lists.openembedded.org
<danielwa=cisco.com@lists.openembedded.org> wrote:

> Adds a option to the git fetcher which can be used to specify a local
> reference for a given SRC_URI. Which is then fed into the clone command
> as "--reference-if-able" which make the option fault tolerant if the
> directory doesn't exist.
>

We tried this in the past (maybe 10 years ago now) to chunk the
kernel into different sets of blobs and do a set of references for
optimal sharing, repacking, etc.

In the end, it was discarded as unnecessary complexity.

I'm still of the opinion that is the case. What problem is this solving
that justifies the complexity ?

the patch doesn't show a real world use case, what the SRC_URI
would look like, where the referenced clone would be, what would
happen if the referenced repository was somehow cleaned, or
any number of different interactions (what happens if shallow is
specified, etc, etc).

Bruce


>
> AI-Generated: OpenAI Codex
> Signed-off-by: Daniel Walker <danielwa@cisco.com>
> Change-Id: Ib5eb685ddfb48fa19f1c65cfc468894806f99e83
> ---
>  .../bitbake-user-manual-fetching.rst                 |  7 +++++++
>  lib/bb/fetch2/git.py                                 | 12 +++++++++++-
>  2 files changed, 18 insertions(+), 1 deletion(-)
>
> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
> b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
> index b96018a4e..6e4fc0800 100644
> --- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
> +++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
> @@ -398,6 +398,13 @@ This fetcher supports the following parameters:
>     preserve the local cache carefully for future use. The default value
>     for this parameter is "0".
>
> +-  *"reference":* Specifies a local directory to use as a reference
> +   repository. When set, the fetcher passes the directory to ``git clone``
> +   with ``--reference-if-able``. Git uses the reference repository's
> +   objects when possible, reducing the amount of object data copied or
> +   fetched. If the reference directory cannot be used, Git continues
> +   cloning without it.
> +
>  -  *"nobranch":* Tells the fetcher to not check the SHA validation for
>     the branch when set to "1". The default is "0". Set this option for
>     the recipe that refers to the commit that is valid for any namespace
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 0fbd85441..df1c4415f 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -22,6 +22,10 @@ Supported SRC_URI options are:
>     remind fetcher to preserve local cache carefully for future use.
>     The default value is "0", set rebaseable=1 for rebaseable git repo.
>
> +- reference
> +   Use the specified local directory as a reference repository by passing
> it
> +   to git clone with --reference-if-able.
> +
>  - nocheckout
>     Don't checkout source code when unpacking. set this option for the
> recipe
>     who has its own routine to checkout code.
> @@ -195,6 +199,12 @@ class Git(FetchMethod):
>          if ud.bareclone:
>              ud.cloneflags.append("--mirror")
>
> +        ud.reference_args = []
> +        ud.reference = ud.parm.get("reference")
> +        if ud.reference:
> +            ud.reference_args = ["--reference-if-able", ud.reference]
> +            ud.cloneflags.extend(ud.reference_args)
> +
>          ud.shallow_skip_fast = False
>          ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1"
>          ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or
> "").split()
> @@ -436,7 +446,7 @@ class Git(FetchMethod):
>                  objects = os.path.join(repourl_path, 'objects')
>                  if os.path.isdir(objects) and not os.path.islink(objects):
>                      repourl = repourl_path
> -            clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror',
> repourl, ud.clonedir, '--progress']
> +            clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror'] +
> ud.reference_args + [repourl, ud.clonedir, '--progress']
>              if ud.proto.lower() != 'file':
>                  bb.fetch2.check_network_access(d, clone_cmd, ud.url)
>              progresshandler = GitProgressHandler(d)
> --
> 2.44.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#19794):
> https://lists.openembedded.org/g/bitbake-devel/message/19794
> Mute This Topic: https://lists.openembedded.org/mt/119994628/1050810
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 7755 bytes --]

Re: [bitbake-devel] [PATCH] fetch/git: add reference option

[Daniel Walker (danielwa)]

On Fri, Jun 26, 2026 at 03:56:02PM -0400, Bruce Ashfield wrote:
> On Fri, Jun 26, 2026 at 3:10 PM Daniel Walker via lists.openembedded.org
> <danielwa=cisco.com@lists.openembedded.org> wrote:
> 
> > Adds a option to the git fetcher which can be used to specify a local
> > reference for a given SRC_URI. Which is then fed into the clone command
> > as "--reference-if-able" which make the option fault tolerant if the
> > directory doesn't exist.
> >
> 
> We tried this in the past (maybe 10 years ago now) to chunk the
> kernel into different sets of blobs and do a set of references for
> optimal sharing, repacking, etc.
> 
> In the end, it was discarded as unnecessary complexity.
> 
> I'm still of the opinion that is the case. What problem is this solving
> that justifies the complexity ?
> 
> the patch doesn't show a real world use case, what the SRC_URI
> would look like, where the referenced clone would be, what would
> happen if the referenced repository was somehow cleaned, or
> any number of different interactions (what happens if shallow is
> specified, etc, etc).
> 

At Cisco we have settings which modify the SRC_URI to become the local repo.

For example,

KERNEL_REPO:pn-linux-cisco = "git:///local/git/cache/;protocol=file"

This works to use a local cache of the repo, but it messes up the state cache so
objects can't be used by others.

If we use "--reference-if-able" then the git host and location remains the same, but the
local copy can speed up the clone.

For down stream users, like Cisco, who work in a complex environment it's
helpful to have flexibility.

The reference is not harmful that I know of, it's only used if it can be used.

Daniel