[PATCH 08/21] bitbake: fetch2/git: allow pack references within download dir

["Paul Gortmaker" <paul.gortmaker@windriver.com>]

Borrowing from "git pack-objects" manpage[1] we have:

   A packed archive is an efficient way to transfer a set of objects
   between two repositories as well as an access efficient archival
   format.  In a packed archive, an object is either stored as a
   compressed whole or as a difference from some other object. The
   latter is often called a delta.

   The packed archive format (.pack) is designed to be self-contained
   so that it can be unpacked without any further information. Therefore,
   each object that a delta depends upon must be present within the pack.

   A pack index file (.idx) is generated for fast, random access to
   the objects in the pack. Placing both the index file (.idx) and the
   packed archive (.pack) in the pack/ subdirectory [...] enables Git
   to read from the pack archive.

As stated, they are self-contained and can encapsulate an entire chunk
of development history starting at point A and ending at point B, which
makes them ideal not just for transferring, but also for sharing that
chunk of history between repositories with common ancestry.

When you ask a server for something, it sends a single pack, because
that is most efficient.  So, using the kernel as an example, lets say we
decide we want to use the 2005 git "epoch" as "A" and the v5.10 tag of a
recent release as "B" to delineate an initial block of history.  We can
grab that chunk of history with a simple:

git clone --bare --single-branch --branch v5.10 git://someserver/linux my-510-lib

and we now have all the history up to and including v5.10 tag in a repo
that we can use with further "clone --reference my-510-lib newstuff" as
a static unchanging library of all those objects for "newstuff" to use.

In making such a clone, Git stores the my-510-lib dir in the
objects/info/alternates of "newstuff" and all the refs and tags carry
forard into [packed]refs of "newstuff".  Subsequent uses of the
"newstuff" repo are only using the objects from "my-510-lib" and they
are conveniently all stored in one pack, as there are no loose objects.
In fact it is a waste of time even having git operations look for them.

This means we can simply remove the alternates file and link the pack
(and accompanying index file) into the pack dir of "newstuff".  And by
using symlinks with relative paths, we get a mirror friendly reference
with no data download duplication.  The "clone --reference" is simply
used as a convenience tool to do 99% of the setup work.  The linked in
pack is "just a normal pack" to the consumer of the commit library.

As such, we introduce a "packref" as a useful parallel addition to the
more well known "altref".  It is ideally suited to reference from static
repositories containing a known block of commit history.  Note that we
don't limit ourselves to a single pack, so that a packref can itself be
using another repo as its own packref and so on.  Hence there is no
arbitrary limit of a depth of 5 as there is with a reference using the
more common alternates (i.e. "altref") method.

This opens the door to insert an arbitrary number of split points between
A and B to fragment large repos, if desired.  An example that shows how
the bulk of kernel history can be split in two, three or four chunks
will be provided for recipes-kernel/linux subsequently.

[1] https://git-scm.com/docs/git-pack-objects

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 .../bitbake-user-manual-fetching.rst          |  7 +++++
 bitbake/lib/bb/fetch2/git.py                  | 31 ++++++++++++++++++-
 2 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 12a68accc2c5..db1d108f4736 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -453,6 +453,13 @@ This fetcher supports the following parameters:
    is responsible (via fetch dependency) for ensuring the reference is
    populated/cloned prior to being called on to act as a reference.
 
+-  *"packref":* Enables cloned ``git://`` URLs to use packs from the named
+   peer from the downloads dir as if they were their own packs of commit
+   history.  Similar to ``altref`` above, but suited for bare static repos
+   for which no loose objects outside of packs will ever be present.
+   Recipe is responsible (via fetch dependency) for ensuring the reference
+   is populated/cloned prior to being called on to act as a reference.
+
 Here are some example URLs: ::
 
    SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 3b0c068bc020..8aec12df2bf8 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -65,6 +65,14 @@ Supported SRC_URI options are:
    operations.  Recipe is responsible (via fetch dependency) for ensuring the
    reference is populated/cloned prior to being called on to act as a reference.
 
+- packref
+   Repository is to use packs from named peer from downloads dir by linking
+   them into objects/pack/ of the new repository as if they were their own
+   packs of commit history.  Similar to altref above, but suited for bare
+   static repos for which no loose objects outside of packs will be present.
+   Recipe is responsible (via fetch dependency) for ensuring the reference
+   is populated/cloned prior to being called on to act as a reference.
+
 - usehead
    For local git:// urls to use the current branch HEAD as the revision for use with
    AUTOREV. Implies nobranch.
@@ -171,10 +179,15 @@ class Git(FetchMethod):
 
         ud.static = ud.parm.get("static","0") == "1"
 
+        ud.packref = ud.parm.get("packref","")
+
         ud.altref = ud.parm.get("altref","")
 
         ud.dlname = ud.parm.get("dlname","")
 
+        if ud.packref and ud.altref:
+            raise bb.fetch2.ParameterError("Please pick only ONE reference style: alternates OR packs", ud.url)
+
         # usehead implies nobranch
         ud.usehead = ud.parm.get("usehead","0") == "1"
         if ud.usehead:
@@ -373,6 +386,14 @@ class Git(FetchMethod):
         static = self.get_git_config(ud, d, repo, "bitbake.static")
         return (static == "true")
 
+    def create_pack_links(self, refname, refdir, dstdir):
+        dstpkdir = os.path.join(dstdir, 'objects', 'pack')
+        refpkdir = os.path.join(refdir, 'objects', 'pack')
+        for item in os.listdir(refpkdir):
+            dst = os.path.join(dstpkdir, item)
+            src = os.path.join('..', '..', '..', refname, 'objects', 'pack', item)
+            os.symlink(src, dst)
+
     def download(self, ud, d):
         """Fetch url"""
 
@@ -388,7 +409,9 @@ class Git(FetchMethod):
 
         repourl = self._get_repo_url(ud)
 
-        refname = ud.altref
+        # Both style of reference are the same for initial clone operation.
+        # It is the post-clone absolute to relative path fixup that differs.
+        refname = ud.altref or ud.packref
         if refname:
             alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
             gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2")
@@ -419,6 +442,12 @@ class Git(FetchMethod):
             if ud.static:
                 runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
 
+            if ud.packref:
+                if not self.repo_is_static(ud, d, refdir):
+                    raise bb.fetch2.FetchError("Pack reference '%s' is not marked as static content." % refname)
+                self.create_pack_links(refname, refdir, ud.clonedir)
+                os.remove(alts)
+
         # Update the checkout if needed
         if self.clonedir_need_update(ud, d):
             if ud.altref:
-- 
2.25.1