[PATCH 06/21] bitbake: fetch2/git: allow alt references within download dir

["Paul Gortmaker" <paul.gortmaker@windriver.com>]

It is no secret that we've needed to formalize some kind of sensible
sharing between git repos with common ancestry.  The opportunity for
sharing the mainline kernel base between linux-yocto and linux-yocto-dev
is the main obvious candidate.

Many of us have long been doing local hacks to pre-populate the
downloads dir and avoid lengthy/large downloads, and some have even
been kind enough to try and get a variant of that upstream[1].

But the problem is that it isn't as simple as adding a new "--reference"
switch to the fetcher and then walking away.  That will result in
non-portable and incompatible use cases with existing work flows.   We
have to put some thought into how we see this fitting into the larger
framework of things and in what ways we want to support use of it.

For example, we know we want relative paths in anything in the download
dir, but "clone --reference" will put an absolute path in the alternates
file by default.  We also don't want the SRC_URI to be containing any
absolute paths either.  We also want (pre)mirror stuff to keep working.

So we set some basic ground rules.  Firstly we don't even expose the
obvious "--reference" argument that some of us are used to at all.  That
is because we are going to have two types of references - the common one
based on objets/info/alternates (this commit) and a pack based one (the
subsequent commit).

We also restrict any references to be peers of the other content in
downloads - no /home/fred/super-SDK/kernel-4.19-rt/ type stuff.  The
idea is that once you factor out the common building blocks of source,
then your kernel (staying with that example) becomes a very small repo.
For example - all of the linux-yocto BSPs (standard and -rt) can be held
in a repo that is only 20-30 megabytes - small enough to attach to email.

Clones via alternates have some limitations - for example there is an
arbitrary limit of six[2] levels of nesting, so a pack reference is
introduced to resolve this subsequently.  However alternaltes make sense
for shared non-static (i.e. WIP repos) endpoints, like stable and -rt.

[1] https://lists.openembedded.org/g/bitbake-devel/topic/77645766
[2] https://github.com/git/git/commit/c2f493a4  # if (depth > 5)

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 .../bitbake-user-manual-fetching.rst          |  6 +++++
 bitbake/lib/bb/fetch2/git.py                  | 24 +++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 94f4cf3363f6..12a68accc2c5 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -447,6 +447,12 @@ This fetcher supports the following parameters:
    parameter implies no branch and only works when the transfer protocol
    is ``file://``.
 
+-  *"altref":* Enables cloned ``git://`` URLs to use the named peer from
+   the downloads dir as a reference through the standard git alternates
+   file for clone (see ``-reference``) and subseqent operations.  Recipe
+   is responsible (via fetch dependency) for ensuring the reference is
+   populated/cloned prior to being called on to act as a reference.
+
 Here are some example URLs: ::
 
    SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 6cd5d09b2ded..e742010229c3 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -59,6 +59,12 @@ Supported SRC_URI options are:
    Once cloned, nothing will change, so optimize accordingly.  The default
    is "0", set static=1 when appropriate.
 
+- altref
+   Repository is to use named peer from downloads dir as a reference through
+   the standard git objects/info/alternates file for clone and subseqent
+   operations.  Recipe is responsible (via fetch dependency) for ensuring the
+   reference is populated/cloned prior to being called on to act as a reference.
+
 - usehead
    For local git:// urls to use the current branch HEAD as the revision for use with
    AUTOREV. Implies nobranch.
@@ -165,6 +171,8 @@ class Git(FetchMethod):
 
         ud.static = ud.parm.get("static","0") == "1"
 
+        ud.altref = ud.parm.get("altref","")
+
         ud.dlname = ud.parm.get("dlname","")
 
         # usehead implies nobranch
@@ -380,9 +388,20 @@ class Git(FetchMethod):
 
         repourl = self._get_repo_url(ud)
 
+        refname = ud.altref
+        if refname:
+            alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
+            gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2")
+            refdir = os.path.join(gitdir, refname)
+            if not os.path.exists(refdir):
+                raise bb.fetch2.FetchError("Unable to find reference %s in %s" % (refname, gitdir))
+
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
             extcloneargs = d.getVar('GITCLONEARGS_' + ud.names[0]) or d.getVar('GITCLONEARGS') or "--bare --mirror"
+            if refname:
+                extcloneargs += " --reference %s" % refdir
+
             # We do this since git will use a "-l" option automatically for local urls where possible
             if repourl.startswith("file://"):
                 repourl = repourl[7:]
@@ -392,6 +411,11 @@ class Git(FetchMethod):
             progresshandler = GitProgressHandler(d)
             runfetchcmd(clone_cmd, d, log=progresshandler)
 
+            # Fix up reference to be relative for tarball/mirror portability
+            if ud.altref:
+                with open(alts, "w") as f:
+                    f.write("../../%s/objects\n" % ud.altref)
+
             if ud.static:
                 runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
 
-- 
2.25.1