Shallow clones with explicit history cutoff?

Shallow clones with explicit history cutoff?

[Matthias Urlichs]

Hi,

use case: I am packaging the FOO program for Debian. FOO is maintained in
git but it has a bunch of problems (e.g. because somebody mistakenly checked
in a huge blob which would give the ).

The current workflow for this is to create a new branch, remove the
offending bits if necessary, create a FOO-clean.tar.xz file, and ship that
as "original source". I find that to be suboptimal.

What I would like to have, instead, is a version of shallow cloning which
cuts off not at a pre-determined depth, but at a given branch (or set of
branches). In other words, given

            +-J--K  (packaged)
           /    /
  +-F--G--H----I    (clean)
 /       /
A---B---C---D---E   (upstream)

a command "git clone --shallow-until upstream $REPO" (or however that would
be named) would create a shallow git archive which contains branches
packaged+clean, with commits FGHIJK. In contrast, with --single-branch and
--depth 4 I would get CGHIJK, which isn't what I'd want.

As I have not spent too much time with the git sources lately (as in "None
at all"), some pointers where to start implementing this would be
appreciated, assuming (a) this has a reasonable chance of landing in git and
(b) nobody beats me to it. ;-)

-- 
-- Matthias Urlichs

Re: Shallow clones with explicit history cutoff?

[Duy Nguyen]

On Thu, Aug 21, 2014 at 10:39 PM, Matthias Urlichs <matthias@urlichs.de> wrote:
> What I would like to have, instead, is a version of shallow cloning which
> cuts off not at a pre-determined depth, but at a given branch (or set of
> branches). In other words, given
>
>             +-J--K  (packaged)
>            /    /
>   +-F--G--H----I    (clean)
>  /       /
> A---B---C---D---E   (upstream)
>
> a command "git clone --shallow-until upstream $REPO" (or however that would
> be named) would create a shallow git archive which contains branches
> packaged+clean, with commits FGHIJK. In contrast, with --single-branch and
> --depth 4 I would get CGHIJK, which isn't what I'd want.

I would imagine a more generic mechanism "git clone
--shallow-rev=<rev> $REPO" where you could pass anything that "git
rev-list" can accept (maybe more restricted, and some verification
required). --shallow-rev could be repeated. So in your case it could
be "git clone --shallow-rev="^A" $REPO". We could even maybe turn
--depth into a generic thing that is accepted by rev-list so that it
could be easily combined with other rev-list options (--shallow-rev
and --depth are mutually exclusive).

> As I have not spent too much time with the git sources lately (as in "None
> at all"), some pointers where to start implementing this would be
> appreciated, assuming (a) this has a reasonable chance of landing in git and
> (b) nobody beats me to it. ;-)

I'd like to see this implemented. You are not the first one
complaining about the (lack of) flexibility of --depth. If you have
time, I may be able to support (I should not take on another topic
given my many ongoing/unfinished topics). The starting point is
upload-pack.c. And GIT_TRACE env variable will be your friend. Search
for get_shallow_commits(). There the function is supposed to traverse
down from want_obj and set/unset SHALLOW/NOT_SHALLOW flags  properly.

SHALLOW flag should be set right before the cut-out commit (e.g. B and
F if you want to cut A out). NOT_SHALLOW flags could be used to remove
shallow lines in the receiver repo. If you traverse past an existing
shallow point in the client (this is the fetch/pull case, not clone),
then you should set NOT_SHALLOW so the client knows to remove that
point from their $GIT_DIR/shallow. Once you set these properly, the
rest should work.
-- 
Duy

Re: Shallow clones with explicit history cutoff?

[Matthias Urlichs]

Hi,

Duy Nguyen:
> On Thu, Aug 21, 2014 at 10:39 PM, Matthias Urlichs <matthias@urlichs.de> wrote:
> > What I would like to have, instead, is a version of shallow cloning which
> > cuts off not at a pre-determined depth, but at a given branch (or set of
> > branches). In other words, given
> >
> >             +-J--K  (packaged)
> >            /    /
> >   +-F--G--H----I    (clean)
> >  /       /
> > A---B---C---D---E   (upstream)
> >
> > a command "git clone --shallow-until upstream $REPO" (or however that would
> > be named) would create a shallow git archive which contains branches
> > packaged+clean, with commits FGHIJK. In contrast, with --single-branch and
> > --depth 4 I would get CGHIJK, which isn't what I'd want.
> 
> I would imagine a more generic mechanism "git clone
> --shallow-rev=<rev> $REPO" where you could pass anything that "git
> rev-list" can accept (maybe more restricted, and some verification
> required). --shallow-rev could be repeated. So in your case it could
> be "git clone --shallow-rev="^A" $REPO".

Umm, no. ^E (or ^upstream) would do what I want. Hopefully. ;-)

But you're right, that would fit far better into the existing git
paradigms.

> > As I have not spent too much time with the git sources lately (as in "None
> > at all"), some pointers where to start implementing this would be
> > appreciated, assuming (a) this has a reasonable chance of landing in git and
> > (b) nobody beats me to it. ;-)
> 
> I'd like to see this implemented. You are not the first one
> complaining about the (lack of) flexibility of --depth. If you have
> time, I may be able to support (I should not take on another topic
> given my many ongoing/unfinished topics).

Welcome to the club. :-/

Thanks for the pointers. I'll see what I can do (and when).

-- 
-- Matthias Urlichs