[RFC PATCH 0/5] Implement git shallow mirror tarball support

[RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

This only implements support for shallow mirror tarballs, not shallow clones.

The mirror tarball filename includes branch, revision, and depth. To enable,
use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
URLs. This variable can hold either a clone depth (e.g. `1` to just get
SRCREV), or a ref or commit, in which case we keep history up to that commit.

The shallow support will not, at this time, function correctly for
linux-yocto, due to its branching scheme & validation.

Example:

    BB_GIT_SHALLOW ?= "1"
    BB_GIT_SHALLOW_pn-linux-yocto = ""
    BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
    BB_GIT_SHALLOW_pn-testrepo = "testbranch"

This implementation will attempt to fetch a full mirror tarball if it was
unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
`BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
tarball will be emitted.


The following changes since commit 0c419902e5f58dcee9b7fbe9af1928da37e18399:

  toaster: reduced amount of instance attributes (2015-08-10 13:56:03 -0700)

are available in the git repository at:

  git@github.com:kergoth/bitbake shallow-git-mirror-tarballs

for you to fetch changes up to 01c1164fe4b54d5f3de1fd4863808af88f06c144:

  bb.fetch.git: use ud.mirrortarballs (2015-08-13 15:48:01 -0700)

----------------------------------------------------------------
Christopher Larson (5):
  bb.fetch.git: add support for shallow mirror tarballs
  bb.fetch.git: truncate branches to SRCREV
  bb.fetch: simplify mirror tarball handling in try_mirror_url
  bb.fetch: support ud.mirrortarballs
  bb.fetch.git: use ud.mirrortarballs

 lib/bb/fetch2/__init__.py |  27 ++++---
 lib/bb/fetch2/git.py      | 182 +++++++++++++++++++++++++++++++++++-----------
 2 files changed, 154 insertions(+), 55 deletions(-)

-- 
2.2.1

[RFC PATCH 1/5] bb.fetch.git: add support for shallow mirror tarballs

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

The mirror tarball filename includes branch, revision, and depth. To enable,
use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
URLs. This variable can hold either a clone depth (e.g. `1` to just get
SRCREV), or a ref or commit, in which case we keep history up to that commit.

The shallow support will not, at this time, function correctly for
linux-yocto, due to its branching scheme & validation.

Example:

    BB_GIT_SHALLOW ?= "1"
    BB_GIT_SHALLOW_pn-linux-yocto = ""
    BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
    BB_GIT_SHALLOW_pn-testrepo = "testbranch"

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 lib/bb/fetch2/git.py | 165 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 125 insertions(+), 40 deletions(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 374d846..e88736d 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -66,8 +66,10 @@ Supported SRC_URI options are:
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
+import itertools
 import os
 import re
+import tempfile
 import bb
 from   bb    import data
 from   bb.fetch2 import FetchMethod
@@ -118,12 +120,24 @@ class Git(FetchMethod):
         branches = ud.parm.get("branch", "master").split(',')
         if len(branches) != len(ud.names):
             raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)
+
+        shallow_default = d.getVar("BB_GIT_SHALLOW", True)
+        ud.shallows = {}
         ud.branches = {}
-        for name in ud.names:
-            branch = branches[ud.names.index(name)]
+        for pos, name in enumerate(ud.names):
+            branch = branches[pos]
             ud.branches[name] = branch
             ud.unresolvedrev[name] = branch
 
+            shallow = d.getVar("BB_GIT_SHALLOW_%s" % name, True) or shallow_default
+            if shallow == "0":
+                shallow = None
+            ud.shallows[name] = shallow
+
+        if not shallow_default and not filter(None, ud.shallows.itervalues()):
+            # No shallows
+            ud.shallows = None
+
         ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git -c core.fsyncobjectfiles=0"
 
         ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0") or ud.rebaseable
@@ -137,32 +151,44 @@ class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':','.'), ud.path.replace('/', '.').replace('*', '.'))
+        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.'))
         # for rebaseable git repo, it is necessary to keep mirror tar ball
         # per revision, so that even the revision disappears from the
         # upstream repo in the future, the mirror will remain intact and still
         # contains the revision
-        if ud.rebaseable:
+        if ud.shallows:
+            for name, shallow in ud.shallows.iteritems():
+                gitsrcname = '%s_%s_%s' % (gitsrcname, ud.branches[name].replace('/', '.'), ud.revisions[name])
+                if shallow:
+                    gitsrcname = gitsrcname + "_" + shallow
+        elif ud.rebaseable:
             for name in ud.names:
                 gitsrcname = gitsrcname + '_' + ud.revisions[name]
-        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
-        ud.fullmirror = os.path.join(d.getVar("DL_DIR", True), ud.mirrortarball)
-        gitdir = d.getVar("GITDIR", True) or (d.getVar("DL_DIR", True) + "/git2/")
-        ud.clonedir = os.path.join(gitdir, gitsrcname)
 
+        dl_dir = d.getVar("DL_DIR", True)
+        gitdir = d.getVar("GITDIR", True) or (dl_dir + "/git2/")
+        ud.clonedir = os.path.join(gitdir, gitsrcname)
         ud.localfile = ud.clonedir
 
+        ud.mirrortarball = 'git2_%s.tar.gz' % gitsrcname
+        ud.fullmirror = os.path.join(dl_dir, ud.mirrortarball)
+
     def localpath(self, ud, d):
-        return ud.clonedir
+        if ud.shallows and os.path.exists(ud.fullmirror):
+            return ud.fullmirror
+        else:
+            return ud.clonedir
 
     def need_update(self, ud, d):
+        if ud.shallows and os.path.exists(ud.fullmirror):
+            return False
         if not os.path.exists(ud.clonedir):
             return True
         os.chdir(ud.clonedir)
         for name in ud.names:
             if not self._contains_ref(ud, d, name):
                 return True
-        if ud.write_tarballs and not os.path.exists(ud.fullmirror):
+        if ud.write_tarballs and not os.path.exists(ud.fullmirror if not ud.shallows else ud.fullmirror):
             return True
         return False
 
@@ -182,9 +208,14 @@ class Git(FetchMethod):
 
         # If the checkout doesn't exist and the mirror tarball does, extract it
         if not os.path.exists(ud.clonedir) and os.path.exists(ud.fullmirror):
-            bb.utils.mkdirhier(ud.clonedir)
-            os.chdir(ud.clonedir)
-            runfetchcmd("tar -xzf %s" % (ud.fullmirror), d)
+            if ud.shallows:
+                ud.localpath = ud.fullmirror
+                ud.ignore_checksums = True
+                return
+            else:
+                bb.utils.mkdirhier(ud.clonedir)
+                os.chdir(ud.clonedir)
+                runfetchcmd("tar -xzf %s" % ud.fullmirror, d)
 
         repourl = self._get_repo_url(ud)
 
@@ -225,22 +256,68 @@ class Git(FetchMethod):
 
     def build_mirror_data(self, ud, d):
         # Generate a mirror tarball if needed
-        if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+        if ud.shallows:
+            should_write = not os.path.exists(ud.fullmirror)
+        else:
+            should_write = ud.repochanged or not os.path.exists(ud.fullmirror)
+
+        if ud.write_ud.fullmirrors and should_write:
             # it's possible that this symlink points to read-only filesystem with PREMIRROR
             if os.path.islink(ud.fullmirror):
                 os.unlink(ud.fullmirror)
 
-            os.chdir(ud.clonedir)
-            logger.info("Creating tarball of git repository")
-            runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
-            runfetchcmd("touch %s.done" % (ud.fullmirror), d)
+            if ud.shallows:
+                tempdir = tempfile.mkdtemp()
+                shallowclone = os.path.join(tempdir, 'git')
+                os.makedirs(shallowclone)
+                try:
+                    os.chdir(shallowclone)
+                    runfetchcmd("%s init --bare" % ud.basecmd, d)
+                    with open(os.path.join(shallowclone, 'shallow'), 'w') as f:
+                        for name, shallow in ud.shallows.iteritems():
+                            if not shallow:
+                                continue
+                            revision = ud.revisions[name]
+
+                            for rev in [shallow + "^{}", shallow]:
+                                try:
+                                    revision = runfetchcmd("GIT_DIR=%s %s rev-parse %s" % (ud.clonedir, ud.basecmd, rev), d)
+                                except bb.fetch2.FetchError:
+                                    pass
+                                else:
+                                    break
+                            else:
+                                try:
+                                    shallow = int(shallow)
+                                except ValueError:
+                                    raise bb.fetch2.FetchError("Invalid shallow= parameter value '%s' in %s" % (shallow, ud.url))
+                                else:
+                                    revision = runfetchcmd("GIT_DIR=%s %s rev-parse %s~%d" % (ud.clonedir, ud.basecmd, revision, shallow - 1), d)
+
+                            f.write(revision)
+
+                    runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d)
+                    for name, branch in ud.branches.iteritems():
+                        runfetchcmd("%s fetch -a origin %s" % (ud.basecmd, branch), d)
+                    repourl = self._get_repo_url(ud)
+                    runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d)
+                    logger.info("Creating ud.fullmirror of git repository")
+                    runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".")), d)
+                    runfetchcmd("touch %s.done" % ud.fullmirror, d)
+                finally:
+                    bb.utils.remove(tempdir, recurse=True)
+            else:
+                os.chdir(ud.clonedir)
+                logger.info("Creating ud.fullmirror of git repository")
+                runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".")), d)
+                runfetchcmd("touch %s.done" % ud.fullmirror, d)
 
     def unpack(self, ud, destdir, d):
         """ unpack the downloaded src to destdir"""
 
         subdir = ud.parm.get("subpath", "")
         if subdir != "":
-            readpathspec = ":%s" % (subdir)
+            readpathspec = ":%s" % subdir
             def_destsuffix = "%s/" % os.path.basename(subdir.rstrip('/'))
         else:
             readpathspec = ""
@@ -251,27 +328,35 @@ class Git(FetchMethod):
         if os.path.exists(destdir):
             bb.utils.prunedir(destdir)
 
-        cloneflags = "-s -n"
-        if ud.bareclone:
-            cloneflags += " --mirror"
-
-        # Versions of git prior to 1.7.9.2 have issues where foo.git and foo get confused
-        # and you end up with some horrible union of the two when you attempt to clone it
-        # The least invasive workaround seems to be a symlink to the real directory to
-        # fool git into ignoring any .git version that may also be present.
-        #
-        # The issue is fixed in more recent versions of git so we can drop this hack in future
-        # when that version becomes common enough.
-        clonedir = ud.clonedir
-        if not ud.path.endswith(".git"):
-            indirectiondir = destdir[:-1] + ".indirectionsymlink"
-            if os.path.exists(indirectiondir):
-                os.remove(indirectiondir)
-            bb.utils.mkdirhier(os.path.dirname(indirectiondir))
-            os.symlink(ud.clonedir, indirectiondir)
-            clonedir = indirectiondir
-
-        runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, cloneflags, clonedir, destdir), d)
+        if ud.shallows and not os.path.exists(ud.clonedir):
+            gitdir = os.path.join(destdir, '.git')
+            bb.utils.mkdirhier(gitdir)
+            os.chdir(gitdir)
+            runfetchcmd("tar -xzf %s" % ud.fullmirror, d)
+            runfetchcmd("git config core.bare false", d)
+        else:
+            cloneflags = "-s -n"
+            if ud.bareclone:
+                cloneflags += " --mirror"
+
+            # Versions of git prior to 1.7.9.2 have issues where foo.git and foo get confused
+            # and you end up with some horrible union of the two when you attempt to clone it
+            # The least invasive workaround seems to be a symlink to the real directory to
+            # fool git into ignoring any .git version that may also be present.
+            #
+            # The issue is fixed in more recent versions of git so we can drop this hack in future
+            # when that version becomes common enough.
+            clonedir = ud.clonedir
+            if not ud.path.endswith(".git"):
+                indirectiondir = destdir[:-1] + ".indirectionsymlink"
+                if os.path.exists(indirectiondir):
+                    os.remove(indirectiondir)
+                bb.utils.mkdirhier(os.path.dirname(indirectiondir))
+                os.symlink(ud.clonedir, indirectiondir)
+                clonedir = indirectiondir
+
+            runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, cloneflags, clonedir, destdir), d)
+
         os.chdir(destdir)
         repourl = self._get_repo_url(ud)
         runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d)
-- 
2.2.1

[RFC PATCH 2/5] bb.fetch.git: truncate branches to SRCREV

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Rather than keeping the commits between SRCREV and the branch HEAD, point the
remote branch at the SRCREV, so no commits after SRCREV are kept. If the user
needs them, a git fetch will pull them down.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 lib/bb/fetch2/git.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index e88736d..c19b111 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -299,6 +299,11 @@ class Git(FetchMethod):
                     runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d)
                     for name, branch in ud.branches.iteritems():
                         runfetchcmd("%s fetch -a origin %s" % (ud.basecmd, branch), d)
+                        runfetchcmd("%s update-ref refs/remotes/origin/%s %s" % (ud.basecmd, branch, ud.revisions[name]), d)
+                    runfetchcmd("%s gc --prune" % ud.basecmd, d)
+                    runfetchcmd("%s prune-packed" % ud.basecmd, d)
+                    runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
+
                     repourl = self._get_repo_url(ud)
                     runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d)
                     logger.info("Creating ud.fullmirror of git repository")
-- 
2.2.1

[RFC PATCH 3/5] bb.fetch: simplify mirror tarball handling in try_mirror_url

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Rather than checking mirrortarball explicitly, simply check and see if the
fetch method used for the mirror differs from the fetch method used for the
original url. If so, and the localpaths differ, we can trust that further
processing by the original fetcher is necessary.

This will make it easier to add the ability to support multiple mirror
tarballs, to implement shallow git mirror tarball support.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 lib/bb/fetch2/__init__.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 7b4d130..c5e1758 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -933,8 +933,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
         # We may be obtaining a mirror tarball which needs further processing by the real fetcher
         # If that tarball is a local file:// we need to provide a symlink to it
         dldir = ld.getVar("DL_DIR", True)
-        if origud.mirrortarball and os.path.basename(ud.localpath) == os.path.basename(origud.mirrortarball) \
-                and os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
+        if origud.method != ud.method:
             # Create donestamp in old format to avoid triggering a re-download
             bb.utils.mkdirhier(os.path.dirname(ud.donestamp))
             open(ud.donestamp, 'w').close()
@@ -943,7 +942,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
                 os.symlink(ud.localpath, dest)
             if not verify_donestamp(origud, ld) or origud.method.need_update(origud, ld):
                 origud.method.download(origud, ld)
-                if hasattr(origud.method,"build_mirror_data"):
+                if hasattr(origud.method, "build_mirror_data"):
                     origud.method.build_mirror_data(origud, ld)
             return ud.localpath
         # Otherwise the result is a local file:// and we symlink to it
-- 
2.2.1

[RFC PATCH 4/5] bb.fetch: support ud.mirrortarballs

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This lets us support multiple types of mirror tarballs. The main use case for
this at this time is for git, to be able to support both shallow and full git
mirror tarballs.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 lib/bb/fetch2/__init__.py | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index c5e1758..207c4e4 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -429,7 +429,7 @@ def encodeurl(decoded):
 
     return url
 
-def uri_replace(ud, uri_find, uri_replace, replacements, d):
+def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
     if not ud.url or not uri_find or not uri_replace:
         logger.error("uri_replace: passed an undefined value, not replacing")
         return None
@@ -468,9 +468,9 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d):
             if loc == 2:
                 # Handle path manipulations
                 basename = None
-                if uri_decoded[0] != uri_replace_decoded[0] and ud.mirrortarball:
+                if uri_decoded[0] != uri_replace_decoded[0] and mirrortarball:
                     # If the source and destination url types differ, must be a mirrortarball mapping
-                    basename = os.path.basename(ud.mirrortarball)
+                    basename = os.path.basename(mirrortarball)
                     # Kill parameters, they make no sense for mirror tarballs
                     uri_decoded[5] = {}
                 elif ud.localpath and ud.method.supports_checksum(ud):
@@ -858,13 +858,13 @@ def build_mirroruris(origud, mirrors, ld):
     replacements["BASENAME"] = origud.path.split("/")[-1]
     replacements["MIRRORNAME"] = origud.host.replace(':','.') + origud.path.replace('/', '.').replace('*', '.')
 
-    def adduri(ud, uris, uds):
+    def adduri(ud, uris, uds, tarball):
         for line in mirrors:
             try:
                 (find, replace) = line
             except ValueError:
                 continue
-            newuri = uri_replace(ud, find, replace, replacements, ld)
+            newuri = uri_replace(ud, find, replace, replacements, ld, tarball)
             if not newuri or newuri in uris or newuri == origud.url:
                 continue
 
@@ -881,16 +881,20 @@ def build_mirroruris(origud, mirrors, ld):
                 try:
                     # setup_localpath of file:// urls may fail, we should still see 
                     # if mirrors of the url exist
-                    adduri(newud, uris, uds)
+                    adduri(newud, uris, uds, tarball)
                 except UnboundLocalError:
                     pass
-                continue   
+                continue
             uris.append(newuri)
             uds.append(newud)
 
-            adduri(newud, uris, uds)
+            adduri(newud, uris, uds, tarball)
 
-    adduri(origud, uris, uds)
+    if hasattr(origud, 'mirrortarballs'):
+        for tarball in origud.mirrortarballs:
+            adduri(origud, uris, uds, tarball)
+    else:
+        adduri(origud, uris, uds, getattr(origud, 'mirrortarball', None))
 
     return uris, uds
 
-- 
2.2.1

[RFC PATCH 5/5] bb.fetch.git: use ud.mirrortarballs

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

If the shallow tarball isn't available, fall back to the full tarball.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 lib/bb/fetch2/git.py | 56 +++++++++++++++++++++++++++++-----------------------
 1 file changed, 31 insertions(+), 25 deletions(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index c19b111..cb2e9a7 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -156,12 +156,7 @@ class Git(FetchMethod):
         # per revision, so that even the revision disappears from the
         # upstream repo in the future, the mirror will remain intact and still
         # contains the revision
-        if ud.shallows:
-            for name, shallow in ud.shallows.iteritems():
-                gitsrcname = '%s_%s_%s' % (gitsrcname, ud.branches[name].replace('/', '.'), ud.revisions[name])
-                if shallow:
-                    gitsrcname = gitsrcname + "_" + shallow
-        elif ud.rebaseable:
+        if ud.rebaseable:
             for name in ud.names:
                 gitsrcname = gitsrcname + '_' + ud.revisions[name]
 
@@ -172,15 +167,24 @@ class Git(FetchMethod):
 
         ud.mirrortarball = 'git2_%s.tar.gz' % gitsrcname
         ud.fullmirror = os.path.join(dl_dir, ud.mirrortarball)
+        if ud.shallows:
+            tarballname = gitsrcname
+            for name, shallow in ud.shallows.iteritems():
+                tarballname = '%s_%s_%s' % (tarballname, ud.branches[name].replace('/', '.'), ud.revisions[name])
+                if shallow:
+                    tarballname = tarballname + "_" + shallow
+            ud.shallowtarball = 'git2_%s.tar.gz' % tarballname
+            ud.fullshallow = os.path.join(dl_dir, ud.shallowtarball)
+            ud.mirrortarballs = [ud.shallowtarball, ud.mirrortarball]
 
     def localpath(self, ud, d):
-        if ud.shallows and os.path.exists(ud.fullmirror):
-            return ud.fullmirror
+        if ud.shallows and os.path.exists(ud.fullshallow):
+            return ud.fullshallow
         else:
             return ud.clonedir
 
     def need_update(self, ud, d):
-        if ud.shallows and os.path.exists(ud.fullmirror):
+        if ud.shallows and os.path.exists(ud.fullshallow):
             return False
         if not os.path.exists(ud.clonedir):
             return True
@@ -188,7 +192,7 @@ class Git(FetchMethod):
         for name in ud.names:
             if not self._contains_ref(ud, d, name):
                 return True
-        if ud.write_tarballs and not os.path.exists(ud.fullmirror if not ud.shallows else ud.fullmirror):
+        if ud.write_tarballs and not os.path.exists(ud.fullmirror if not ud.shallows else ud.fullshallow):
             return True
         return False
 
@@ -207,12 +211,12 @@ class Git(FetchMethod):
         ud.repochanged = not os.path.exists(ud.fullmirror)
 
         # If the checkout doesn't exist and the mirror tarball does, extract it
-        if not os.path.exists(ud.clonedir) and os.path.exists(ud.fullmirror):
-            if ud.shallows:
-                ud.localpath = ud.fullmirror
+        if not os.path.exists(ud.clonedir):
+            if ud.shallows and os.path.exists(ud.fullshallow):
+                ud.localpath = ud.fullshallow
                 ud.ignore_checksums = True
                 return
-            else:
+            elif os.path.exists(ud.fullmirror):
                 bb.utils.mkdirhier(ud.clonedir)
                 os.chdir(ud.clonedir)
                 runfetchcmd("tar -xzf %s" % ud.fullmirror, d)
@@ -257,14 +261,16 @@ class Git(FetchMethod):
     def build_mirror_data(self, ud, d):
         # Generate a mirror tarball if needed
         if ud.shallows:
-            should_write = not os.path.exists(ud.fullmirror)
+            tarball = ud.fullshallow
+            should_write = not os.path.exists(ud.fullshallow)
         else:
+            tarball = ud.fullmirror
             should_write = ud.repochanged or not os.path.exists(ud.fullmirror)
 
-        if ud.write_ud.fullmirrors and should_write:
+        if ud.write_tarballs and should_write:
             # it's possible that this symlink points to read-only filesystem with PREMIRROR
-            if os.path.islink(ud.fullmirror):
-                os.unlink(ud.fullmirror)
+            if os.path.islink(tarball):
+                os.unlink(tarball)
 
             if ud.shallows:
                 tempdir = tempfile.mkdtemp()
@@ -306,16 +312,16 @@ class Git(FetchMethod):
 
                     repourl = self._get_repo_url(ud)
                     runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d)
-                    logger.info("Creating ud.fullmirror of git repository")
-                    runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".")), d)
-                    runfetchcmd("touch %s.done" % ud.fullmirror, d)
+                    logger.info("Creating tarball of git repository")
+                    runfetchcmd("tar -czf %s %s" % (tarball, os.path.join(".")), d)
+                    runfetchcmd("touch %s.done" % tarball, d)
                 finally:
                     bb.utils.remove(tempdir, recurse=True)
             else:
                 os.chdir(ud.clonedir)
-                logger.info("Creating ud.fullmirror of git repository")
-                runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".")), d)
-                runfetchcmd("touch %s.done" % ud.fullmirror, d)
+                logger.info("Creating tarball of git repository")
+                runfetchcmd("tar -czf %s %s" % (tarball, os.path.join(".")), d)
+                runfetchcmd("touch %s.done" % tarball, d)
 
     def unpack(self, ud, destdir, d):
         """ unpack the downloaded src to destdir"""
@@ -337,7 +343,7 @@ class Git(FetchMethod):
             gitdir = os.path.join(destdir, '.git')
             bb.utils.mkdirhier(gitdir)
             os.chdir(gitdir)
-            runfetchcmd("tar -xzf %s" % ud.fullmirror, d)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d)
             runfetchcmd("git config core.bare false", d)
         else:
             cloneflags = "-s -n"
-- 
2.2.1

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 4552 bytes --]

Some more notes on this implementation:

On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> This only implements support for shallow mirror tarballs, not shallow
> clones.
>

Shallow clones from upstream are not implemented, as, first, you can't
shallow clone from a commit, only the HEAD of a branch, and second, we'd
have to hardcode the exact depth from branch HEAD to SRCREV (this is in
addition to clone depth beyond SRCREV which is covered by this patch
series).

The mirror tarball filename includes branch, revision, and depth. To enable,
> use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
> URLs. This variable can hold either a clone depth (e.g. `1` to just get
> SRCREV), or a ref or commit, in which case we keep history up to that
> commit.
>
> The shallow support will not, at this time, function correctly for
> linux-yocto, due to its branching scheme & validation.
>

It'd be possible to support linux-yocto, but only if all the dependent
branches were called out in the URI, so they all get retained.

Example:
>
>     BB_GIT_SHALLOW ?= "1"
>     BB_GIT_SHALLOW_pn-linux-yocto = ""
>     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>
> This implementation will attempt to fetch a full mirror tarball if it was
> unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
> tarball will be emitted.
>

I'm not 100% happy with this implementation, at the very least it needs a
little refactoring, to split out a new function or two, and make it a bit
more pythonic, but it might not even be the right approach, which is why
it's an RFC.

The shallow depth control is a variable rather than a URL parameter because
I did the latter initially before realizing that obviously making it a url
parameter would change checksums and cause pointless rebuilds, so the
variable is the best approach there. Opinions on the name of the variable
are welcome. Also open to thoughts on BB_GIT_SHALLOW_<name> vs
BB_GIT_SHALLOW[<name>].

I did attempt to prototype a new fetcher which acted as a premirror for the
git fetcher to handle the shallow logic, but ran into a number of issues. I
don't think any of them are unsurmountable, but it was enough to push me
into trying it this way first:

- The premirror fetcher would need to handle the fetching from the mirrors
itself, since the mirror tarball logic isn't nested, it's only used against
the main urldata, not the mirror urldata.
    - I think this would be fairly straightforward, just a matter of
calling try_mirrors() in download(), with appropriate arguments and
metadata.
- The wrong unpack method gets called. We'd need the premirror fetcher's
unpack method to be called, not the main fetcher's unpack method.
    - I think this could be addressed by passing in the mirror urldata into
the main fetcher's download/unpack methods, so it could call the mirror
unpack() to do the real work instead of handling it itself for cases like
this.
- The try_mirror_uri mirrortarball logic would have to remain, to ensure
that the main fetcher's download method doesn't get called in this context,
which is okay, but then we hit an issue with the symlinking that happens
otherwise, which is used to cover file:// mirrors -- we end up with the
shallow tarball symlinked to ${DL_DIR}/git2/<path>.
    - If we pass the mirror urldata into the main fetcher methods,
presumably we could ditch the try_mirror_uri mirrortarball logic the way I
do in this series, and then the main fetcher download method could simply
punt if it doesn't know how to handle the mirror in question (e.g. not a
non-shallow tarball)
    - Regarding the symlinking, I think we could trivially address that by
making sure the localpath basenames are the same for both urls, and if not,
don't symlink.

So we'll need to consider whether it'd be better to try a prototype of that
approach, addressing the issues, or continue along this path.

All that said, this does do the job, and seemingly does it quite well, so
it's possible we could use this and then improve the underlying
implementation as a secondary step (though, admittedly, doing that tends to
result in never coming back to do it better :).
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 6134 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1621 bytes --]

On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> Please review the following changes for suitability for inclusion. If you
> have
> any objections or suggestions for improvement, please respond to the
> patches. If
> you agree with the changes, please provide your Acked-by.
>
> This only implements support for shallow mirror tarballs, not shallow
> clones.
>
> The mirror tarball filename includes branch, revision, and depth. To
> enable,
> use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
> URLs. This variable can hold either a clone depth (e.g. `1` to just get
> SRCREV), or a ref or commit, in which case we keep history up to that
> commit.
>
> The shallow support will not, at this time, function correctly for
> linux-yocto, due to its branching scheme & validation.
>
> Example:
>
>     BB_GIT_SHALLOW ?= "1"
>     BB_GIT_SHALLOW_pn-linux-yocto = ""
>     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>
> This implementation will attempt to fetch a full mirror tarball if it was
> unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
> tarball will be emitted.
>

I forgot to mention, this implements [YOCTO #7958](
https://bugzilla.yoctoproject.org/show_bug.cgi?id=7958).
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2248 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]

On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> This only implements support for shallow mirror tarballs, not shallow
> clones.
>
> The mirror tarball filename includes branch, revision, and depth. To
> enable,
> use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
> URLs. This variable can hold either a clone depth (e.g. `1` to just get
> SRCREV), or a ref or commit, in which case we keep history up to that
> commit.
>
> The shallow support will not, at this time, function correctly for
> linux-yocto, due to its branching scheme & validation.
>
> Example:
>
>     BB_GIT_SHALLOW ?= "1"
>     BB_GIT_SHALLOW_pn-linux-yocto = ""
>     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>
> This implementation will attempt to fetch a full mirror tarball if it was
> unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
> tarball will be emitted.
>

The other thing I forgot to mention is that the shallow tarballs aren't
unpacked into ${DL_DIR}/git2/ the way the full ones are, they're unpacked
directly into the destination under ${WORKDIR}, to avoid complexities and
confusion. This is why localpath() now returns the shallow tarball path
when it exists, to avoid issues with checksums, and because we don't
need/want download() to do anything else after it's fetched.

I just realized there's a slight inconsistency in the current behavior.
When shallow and the shallow tarball exists, localpath() returns that path,
but in the case where both ud.clonedir and the shallow tarball exists, the
former will be unpacked, not the latter. I'll have to double check that the
case where we have an existing out of date clonedir doesn't cause a shallow
tarball to be fetched, otherwise it'll bypass updating the clonedir in
download() and then unpack the out of date clonedir into workdir. If that's
an issue, can make unpack() prefer the shallow tarball when shallow and the
tarball exists, even if clonedir exists.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2881 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2344 bytes --]

On Fri, Aug 14, 2015 at 08:57:17AM -0700, Christopher Larson wrote:
> On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
> wrote:
> 
> > From: Christopher Larson <chris_larson@mentor.com>
> >
> > Please review the following changes for suitability for inclusion. If you
> > have
> > any objections or suggestions for improvement, please respond to the
> > patches. If
> > you agree with the changes, please provide your Acked-by.
> >
> > This only implements support for shallow mirror tarballs, not shallow
> > clones.
> >
> > The mirror tarball filename includes branch, revision, and depth. To
> > enable,
> > use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
> > URLs. This variable can hold either a clone depth (e.g. `1` to just get
> > SRCREV), or a ref or commit, in which case we keep history up to that
> > commit.
> >
> > The shallow support will not, at this time, function correctly for
> > linux-yocto, due to its branching scheme & validation.
> >
> > Example:
> >
> >     BB_GIT_SHALLOW ?= "1"
> >     BB_GIT_SHALLOW_pn-linux-yocto = ""
> >     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
> >     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
> >
> > This implementation will attempt to fetch a full mirror tarball if it was
> > unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> > `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
> > tarball will be emitted.

It would be nice to describe what this implementation does when there is
some local git clone already (will it first try to update it with "git fetch"
or download shallow tarball?)

> >
> 
> I forgot to mention, this implements [YOCTO #7958](
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=7958).

it also looks like more proper fix for [YOCTO #1511](
https://bugzilla.yoctoproject.org/show_bug.cgi?id=1511).

> -- 
> Christopher Larson
> kergoth at gmail dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics

> -- 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel


-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

On Tue, Aug 18, 2015 at 1:45 AM, Martin Jansa <martin.jansa@gmail.com>
wrote:

> On Fri, Aug 14, 2015 at 08:57:17AM -0700, Christopher Larson wrote:
> > On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
> > wrote:
> >
> > > From: Christopher Larson <chris_larson@mentor.com>
> > >
> > > Please review the following changes for suitability for inclusion. If
> you
> > > have
> > > any objections or suggestions for improvement, please respond to the
> > > patches. If
> > > you agree with the changes, please provide your Acked-by.
> > >
> > > This only implements support for shallow mirror tarballs, not shallow
> > > clones.
> > >
> > > The mirror tarball filename includes branch, revision, and depth. To
> > > enable,
> > > use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for
> specific
> > > URLs. This variable can hold either a clone depth (e.g. `1` to just get
> > > SRCREV), or a ref or commit, in which case we keep history up to that
> > > commit.
> > >
> > > The shallow support will not, at this time, function correctly for
> > > linux-yocto, due to its branching scheme & validation.
> > >
> > > Example:
> > >
> > >     BB_GIT_SHALLOW ?= "1"
> > >     BB_GIT_SHALLOW_pn-linux-yocto = ""
> > >     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
> > >     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
> > >
> > > This implementation will attempt to fetch a full mirror tarball if it
> was
> > > unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> > > `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
> > > tarball will be emitted.
>
> It would be nice to describe what this implementation does when there is
> some local git clone already (will it first try to update it with "git
> fetch"
> or download shallow tarball?)


In that case, I believe it will update the clone and use that, unless we
have a current tarball already downloaded, but I'll double check the logic
and get back to you.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3022 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2947 bytes --]

On Tue, Aug 18, 2015 at 7:28 AM, Christopher Larson <kergoth@gmail.com>
wrote:

>
> On Tue, Aug 18, 2015 at 1:45 AM, Martin Jansa <martin.jansa@gmail.com>
> wrote:
>
>> On Fri, Aug 14, 2015 at 08:57:17AM -0700, Christopher Larson wrote:
>> > On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
>> > wrote:
>> >
>> > > From: Christopher Larson <chris_larson@mentor.com>
>> > >
>> > > Please review the following changes for suitability for inclusion. If
>> you
>> > > have
>> > > any objections or suggestions for improvement, please respond to the
>> > > patches. If
>> > > you agree with the changes, please provide your Acked-by.
>> > >
>> > > This only implements support for shallow mirror tarballs, not shallow
>> > > clones.
>> > >
>> > > The mirror tarball filename includes branch, revision, and depth. To
>> > > enable,
>> > > use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for
>> specific
>> > > URLs. This variable can hold either a clone depth (e.g. `1` to just
>> get
>> > > SRCREV), or a ref or commit, in which case we keep history up to that
>> > > commit.
>> > >
>> > > The shallow support will not, at this time, function correctly for
>> > > linux-yocto, due to its branching scheme & validation.
>> > >
>> > > Example:
>> > >
>> > >     BB_GIT_SHALLOW ?= "1"
>> > >     BB_GIT_SHALLOW_pn-linux-yocto = ""
>> > >     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>> > >     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>> > >
>> > > This implementation will attempt to fetch a full mirror tarball if it
>> was
>> > > unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
>> > > `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a
>> shallow
>> > > tarball will be emitted.
>>
>> It would be nice to describe what this implementation does when there is
>> some local git clone already (will it first try to update it with "git
>> fetch"
>> or download shallow tarball?)
>
>
> In that case, I believe it will update the clone and use that, unless we
> have a current tarball already downloaded, but I'll double check the logic
> and get back to you.


I was correct about the behavioral inconsistency of the current
implementation -- if both existed, localpath was returning the mirror
tarball, needs_update was returning false regardless of the state of the
clone, and then unpack would unpack the out of date clone instead of the
tarball :) Not ideal, obviously -- either we prefer a shallow tarball to a
clone or vice versa, not somewhere in-between. I'm fixing that now.

I think that since the clone is essentially a superset (or should be) of
the clone tarball, it should probably be preferred when both are around,
but I'm open to opinions on that.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 4093 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3881 bytes --]

On Tue, Aug 18, 2015 at 11:35:03AM -0700, Christopher Larson wrote:
> On Tue, Aug 18, 2015 at 7:28 AM, Christopher Larson <kergoth@gmail.com>
> wrote:
> 
> >
> > On Tue, Aug 18, 2015 at 1:45 AM, Martin Jansa <martin.jansa@gmail.com>
> > wrote:
> >
> >> On Fri, Aug 14, 2015 at 08:57:17AM -0700, Christopher Larson wrote:
> >> > On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
> >> > wrote:
> >> >
> >> > > From: Christopher Larson <chris_larson@mentor.com>
> >> > >
> >> > > Please review the following changes for suitability for inclusion. If
> >> you
> >> > > have
> >> > > any objections or suggestions for improvement, please respond to the
> >> > > patches. If
> >> > > you agree with the changes, please provide your Acked-by.
> >> > >
> >> > > This only implements support for shallow mirror tarballs, not shallow
> >> > > clones.
> >> > >
> >> > > The mirror tarball filename includes branch, revision, and depth. To
> >> > > enable,
> >> > > use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for
> >> specific
> >> > > URLs. This variable can hold either a clone depth (e.g. `1` to just
> >> get
> >> > > SRCREV), or a ref or commit, in which case we keep history up to that
> >> > > commit.
> >> > >
> >> > > The shallow support will not, at this time, function correctly for
> >> > > linux-yocto, due to its branching scheme & validation.
> >> > >
> >> > > Example:
> >> > >
> >> > >     BB_GIT_SHALLOW ?= "1"
> >> > >     BB_GIT_SHALLOW_pn-linux-yocto = ""
> >> > >     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
> >> > >     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
> >> > >
> >> > > This implementation will attempt to fetch a full mirror tarball if it
> >> was
> >> > > unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> >> > > `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a
> >> shallow
> >> > > tarball will be emitted.
> >>
> >> It would be nice to describe what this implementation does when there is
> >> some local git clone already (will it first try to update it with "git
> >> fetch"
> >> or download shallow tarball?)
> >
> >
> > In that case, I believe it will update the clone and use that, unless we
> > have a current tarball already downloaded, but I'll double check the logic
> > and get back to you.
> 
> 
> I was correct about the behavioral inconsistency of the current
> implementation -- if both existed, localpath was returning the mirror
> tarball, needs_update was returning false regardless of the state of the
> clone, and then unpack would unpack the out of date clone instead of the
> tarball :) Not ideal, obviously -- either we prefer a shallow tarball to a
> clone or vice versa, not somewhere in-between. I'm fixing that now.
> 
> I think that since the clone is essentially a superset (or should be) of
> the clone tarball, it should probably be preferred when both are around,
> but I'm open to opinions on that.

I agree that existing clone should be preferred over shallow tarball
(doing incremental update should be faster in most repositories and
updating local clone will be useful next time SRCREV is upgraded again).

When fetching from premirror shallow tarball should be preferred over
full clone (maybe make it configureable for cases where shallow clone is
still big compared with full clone and when developer expects many
SRCREV bumps - e.g. for AUTOREV - better to fetch full 10GB clone once
per builder machine and do git fetch every 2 days, than downloading 1GB
shallow clone tarball every 2 days - we have this case with huge
*webkit* or chromium repos already).

I think this is matching with Koen's suggestion in:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=1511

Thanks for working on this.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 4890 bytes --]

On Tue, Aug 18, 2015 at 11:51 AM, Martin Jansa <martin.jansa@gmail.com>
wrote:

> On Tue, Aug 18, 2015 at 11:35:03AM -0700, Christopher Larson wrote:
> > On Tue, Aug 18, 2015 at 7:28 AM, Christopher Larson <kergoth@gmail.com>
> > wrote:
> >
> > >
> > > On Tue, Aug 18, 2015 at 1:45 AM, Martin Jansa <martin.jansa@gmail.com>
> > > wrote:
> > >
> > >> On Fri, Aug 14, 2015 at 08:57:17AM -0700, Christopher Larson wrote:
> > >> > On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <
> kergoth@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > From: Christopher Larson <chris_larson@mentor.com>
> > >> > >
> > >> > > Please review the following changes for suitability for
> inclusion. If
> > >> you
> > >> > > have
> > >> > > any objections or suggestions for improvement, please respond to
> the
> > >> > > patches. If
> > >> > > you agree with the changes, please provide your Acked-by.
> > >> > >
> > >> > > This only implements support for shallow mirror tarballs, not
> shallow
> > >> > > clones.
> > >> > >
> > >> > > The mirror tarball filename includes branch, revision, and depth.
> To
> > >> > > enable,
> > >> > > use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for
> > >> specific
> > >> > > URLs. This variable can hold either a clone depth (e.g. `1` to
> just
> > >> get
> > >> > > SRCREV), or a ref or commit, in which case we keep history up to
> that
> > >> > > commit.
> > >> > >
> > >> > > The shallow support will not, at this time, function correctly for
> > >> > > linux-yocto, due to its branching scheme & validation.
> > >> > >
> > >> > > Example:
> > >> > >
> > >> > >     BB_GIT_SHALLOW ?= "1"
> > >> > >     BB_GIT_SHALLOW_pn-linux-yocto = ""
> > >> > >     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
> > >> > >     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
> > >> > >
> > >> > > This implementation will attempt to fetch a full mirror tarball
> if it
> > >> was
> > >> > > unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> > >> > > `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a
> > >> shallow
> > >> > > tarball will be emitted.
> > >>
> > >> It would be nice to describe what this implementation does when there
> is
> > >> some local git clone already (will it first try to update it with "git
> > >> fetch"
> > >> or download shallow tarball?)
> > >
> > >
> > > In that case, I believe it will update the clone and use that, unless
> we
> > > have a current tarball already downloaded, but I'll double check the
> logic
> > > and get back to you.
> >
> >
> > I was correct about the behavioral inconsistency of the current
> > implementation -- if both existed, localpath was returning the mirror
> > tarball, needs_update was returning false regardless of the state of the
> > clone, and then unpack would unpack the out of date clone instead of the
> > tarball :) Not ideal, obviously -- either we prefer a shallow tarball to
> a
> > clone or vice versa, not somewhere in-between. I'm fixing that now.
> >
> > I think that since the clone is essentially a superset (or should be) of
> > the clone tarball, it should probably be preferred when both are around,
> > but I'm open to opinions on that.
>
> I agree that existing clone should be preferred over shallow tarball
> (doing incremental update should be faster in most repositories and
> updating local clone will be useful next time SRCREV is upgraded again).
>

The one concern I have is, what if we have a perfectly useful shallow
tarball, and an out of date clone, and upstream is down, so the fetch to
update it fails. Ideally, the MIRRORS (not PREMIRRORS) would then be
fetched, and it'd realize that the shallow tarball already exists and use
that, but I suspect unpack() would end up using the out of date clone in
that case, since we prefer the clone. I'll have to test that case out and
see if I need to manually check needs_update() in unpack to decide which to
use.

When fetching from premirror shallow tarball should be preferred over
> full clone (maybe make it configureable for cases where shallow clone is
> still big compared with full clone and when developer expects many
> SRCREV bumps - e.g. for AUTOREV - better to fetch full 10GB clone once
> per builder machine and do git fetch every 2 days, than downloading 1GB
> shallow clone tarball every 2 days - we have this case with huge
> *webkit* or chromium repos already).


Yeah, that should be doable with this implementation, either enable it
globally and then disable on a per-recipe basis, or use inline python to
change the value based on AUTOREV usage, or keep it off globally and opt-in
on a per-recipe basis.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 6567 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 4864 bytes --]

On Tue, Aug 18, 2015 at 3:44 PM, Christopher Larson <kergoth@gmail.com>
wrote:

>
> On Tue, Aug 18, 2015 at 11:51 AM, Martin Jansa <martin.jansa@gmail.com>
> wrote:
>
>> On Tue, Aug 18, 2015 at 11:35:03AM -0700, Christopher Larson wrote:
>> > On Tue, Aug 18, 2015 at 7:28 AM, Christopher Larson <kergoth@gmail.com>
>> > wrote:
>> >
>> > >
>> > > On Tue, Aug 18, 2015 at 1:45 AM, Martin Jansa <martin.jansa@gmail.com
>> >
>> > > wrote:
>> > >
>> > >> On Fri, Aug 14, 2015 at 08:57:17AM -0700, Christopher Larson wrote:
>> > >> > On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <
>> kergoth@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > From: Christopher Larson <chris_larson@mentor.com>
>> > >> > >
>> > >> > > Please review the following changes for suitability for
>> inclusion. If
>> > >> you
>> > >> > > have
>> > >> > > any objections or suggestions for improvement, please respond to
>> the
>> > >> > > patches. If
>> > >> > > you agree with the changes, please provide your Acked-by.
>> > >> > >
>> > >> > > This only implements support for shallow mirror tarballs, not
>> shallow
>> > >> > > clones.
>> > >> > >
>> > >> > > The mirror tarball filename includes branch, revision, and
>> depth. To
>> > >> > > enable,
>> > >> > > use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for
>> > >> specific
>> > >> > > URLs. This variable can hold either a clone depth (e.g. `1` to
>> just
>> > >> get
>> > >> > > SRCREV), or a ref or commit, in which case we keep history up to
>> that
>> > >> > > commit.
>> > >> > >
>> > >> > > The shallow support will not, at this time, function correctly
>> for
>> > >> > > linux-yocto, due to its branching scheme & validation.
>> > >> > >
>> > >> > > Example:
>> > >> > >
>> > >> > >     BB_GIT_SHALLOW ?= "1"
>> > >> > >     BB_GIT_SHALLOW_pn-linux-yocto = ""
>> > >> > >     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>> > >> > >     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>> > >> > >
>> > >> > > This implementation will attempt to fetch a full mirror tarball
>> if it
>> > >> was
>> > >> > > unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
>> > >> > > `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a
>> > >> shallow
>> > >> > > tarball will be emitted.
>> > >>
>> > >> It would be nice to describe what this implementation does when
>> there is
>> > >> some local git clone already (will it first try to update it with
>> "git
>> > >> fetch"
>> > >> or download shallow tarball?)
>> > >
>> > >
>> > > In that case, I believe it will update the clone and use that, unless
>> we
>> > > have a current tarball already downloaded, but I'll double check the
>> logic
>> > > and get back to you.
>> >
>> >
>> > I was correct about the behavioral inconsistency of the current
>> > implementation -- if both existed, localpath was returning the mirror
>> > tarball, needs_update was returning false regardless of the state of the
>> > clone, and then unpack would unpack the out of date clone instead of the
>> > tarball :) Not ideal, obviously -- either we prefer a shallow tarball
>> to a
>> > clone or vice versa, not somewhere in-between. I'm fixing that now.
>> >
>> > I think that since the clone is essentially a superset (or should be) of
>> > the clone tarball, it should probably be preferred when both are around,
>> > but I'm open to opinions on that.
>>
>> I agree that existing clone should be preferred over shallow tarball
>> (doing incremental update should be faster in most repositories and
>> updating local clone will be useful next time SRCREV is upgraded again).
>>
>
> The one concern I have is, what if we have a perfectly useful shallow
> tarball, and an out of date clone, and upstream is down, so the fetch to
> update it fails. Ideally, the MIRRORS (not PREMIRRORS) would then be
> fetched, and it'd realize that the shallow tarball already exists and use
> that, but I suspect unpack() would end up using the out of date clone in
> that case, since we prefer the clone. I'll have to test that case out and
> see if I need to manually check needs_update() in unpack to decide which to
> use.
>

FYI, I confirmed that this is handled, though in a slightly surprising way.
If you have an existing out of date clone, and the attempt to update it
fails, the fetcher *removes* the entire local clone and you start fresh.
Not ideal, obviously, and I have a fix for it locally, but in that case the
fallback to mirrors works painlessly, as the out of date clonedir is
removed by the time the mirror fetching and unpack happens. I'll prep
another RFC commit to change that behavior.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 6636 bytes --]

Re: [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> Please review the following changes for suitability for inclusion. If you
> have
> any objections or suggestions for improvement, please respond to the
> patches. If
> you agree with the changes, please provide your Acked-by.
>
> This only implements support for shallow mirror tarballs, not shallow
> clones.
>
> The mirror tarball filename includes branch, revision, and depth. To
> enable,
> use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for specific
> URLs. This variable can hold either a clone depth (e.g. `1` to just get
> SRCREV), or a ref or commit, in which case we keep history up to that
> commit.
>
> The shallow support will not, at this time, function correctly for
> linux-yocto, due to its branching scheme & validation.
>
> Example:
>
>     BB_GIT_SHALLOW ?= "1"
>     BB_GIT_SHALLOW_pn-linux-yocto = ""
>     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>
> This implementation will attempt to fetch a full mirror tarball if it was
> unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
> `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a shallow
> tarball will be emitted.
>

Two new patch series have been submitted, one with general fetch fixes, one
a simplified and cleaned up version of this. I'd greatly appreciate any
review or testers of this, as there are many corner cases to consider in
the fetch code, so the more testers, the better.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2372 bytes --]