Grafts workflow for a "shallow" repository...?

Grafts workflow for a "shallow" repository...?

[Martin Langhoff]

Here is my attempt at a "let's publish a shallow repository for branch
of moodle". Let me show you what I did...

 # before we start
 git --version
 git version 1.5.5.1

 cat /etc/redhat-release
 Fedora release 9 (Sulphur)

 # clone the 'full' repo ~200MB fully packed
 git clone git://git.catalyst.net.nz/moodle-r2.git

 # 1.7 was a significant release, anything earlier than that
 # is just not interesting -- even for pickaxe/annotate purposes
 # so add a graft point right at the branching point.
 cd moodle-r2
 echo `git merge-base origin/cvshead origin/MOODLE_17_STABLE` >>
.git/info/grafts

 # push the branches I care about to a newly created repo on a different server
 # this is approx 50MB... (as opposed to 200MB)
 git push git+ssh://dev.laptop.org/home/martin/public_git/moodle.git \
    origin/MOODLE_19_STABLE:refs/heads/MOODLE_19_STABLE
 git push git+ssh://dev.laptop.org/home/martin/public_git/moodle.git \
    origin/cvshead:refs/heads/cvshead
 # fixed up the server-side repo to have HEAD pointing to a new branch...

 # clone to see it works...
 git clone  git://dev.laptop.org/git/users/martin/moodle.git

 # and now gitk bails out on me. Not entirely unexpected...
 gitk --all

Is this kind of workflow (or a variation of it) supported? For this to
work, we should communicate the grafts in some push operations and
read them in clone ops - and perhaps in fetch too.

cheers,



martin
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: Grafts workflow for a "shallow" repository...?

[Shawn O. Pearce]

Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Here is my attempt at a "let's publish a shallow repository for branch
> of moodle". Let me show you what I did...
...
>  # 1.7 was a significant release, anything earlier than that
>  # is just not interesting -- even for pickaxe/annotate purposes
>  # so add a graft point right at the branching point.
...
> Is this kind of workflow (or a variation of it) supported? For this to
> work, we should communicate the grafts in some push operations and
> read them in clone ops - and perhaps in fetch too.

Currently the grafts file isn't transferred over any transport
protocol as it is considered to be local only to the repository.

For one thing, grafts are a security risk.  Any user can graft
anything in at any position and log/blame operations will honor
the graft, rather than what is stored in the signed commit chain.
Its a low risk, but it allows a peer to lie to you and give you
something other than what you asked for.

Pushing (or fetching) a graft just seems ugly.  You don't want
to fetch a graft if you have no grafts because you have the full
history.  Nor do you want to fetch a graft that might possibly
overwrite/replace a graft you already have.  You might not want to
push all of your available grafts to a remote.  Etc.

I think that in this case the best thing to do is give users
a shell script that does roughly:

	git init
	echo $BASE >.git/info/grafts
	git remote add -f origin $url
	git checkout -b master origin/master

Sign the script, and let users verify it before executing.  You may
also want a script to drag in the history behind by removing the
graft and fetching $BASE^, but that is hard because your repository
already "has" that.

-- 
Shawn.

Re: Grafts workflow for a "shallow" repository...?

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> Martin Langhoff <martin.langhoff@gmail.com> wrote:
>> Here is my attempt at a "let's publish a shallow repository for branch
>> of moodle". Let me show you what I did...
> ...
>>  # 1.7 was a significant release, anything earlier than that
>>  # is just not interesting -- even for pickaxe/annotate purposes
>>  # so add a graft point right at the branching point.
> ...
>> Is this kind of workflow (or a variation of it) supported? For this to
>> work, we should communicate the grafts in some push operations and
>> read them in clone ops - and perhaps in fetch too.
> ...
> I think that in this case the best thing to do is give users
> a shell script that does roughly:
>
> 	git init
> 	echo $BASE >.git/info/grafts
> 	git remote add -f origin $url
> 	git checkout -b master origin/master
>
> Sign the script, and let users verify it before executing.  You may
> also want a script to drag in the history behind by removing the
> graft and fetching $BASE^, but that is hard because your repository
> already "has" that.

Why not just filter-branch _once at the origin_ and publish the result?

Re: Grafts workflow for a "shallow" repository...?

[Björn Steinbrink]

On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:
> "Shawn O. Pearce" <spearce@spearce.org> writes:
> 
> > Martin Langhoff <martin.langhoff@gmail.com> wrote:
> >> Here is my attempt at a "let's publish a shallow repository for branch
> >> of moodle". Let me show you what I did...
> > ...
> >>  # 1.7 was a significant release, anything earlier than that
> >>  # is just not interesting -- even for pickaxe/annotate purposes
> >>  # so add a graft point right at the branching point.
> > ...
> >> Is this kind of workflow (or a variation of it) supported? For this to
> >> work, we should communicate the grafts in some push operations and
> >> read them in clone ops - and perhaps in fetch too.
> > ...
> > I think that in this case the best thing to do is give users
> > a shell script that does roughly:
> >
> > 	git init
> > 	echo $BASE >.git/info/grafts
> > 	git remote add -f origin $url
> > 	git checkout -b master origin/master
> >
> > Sign the script, and let users verify it before executing.  You may
> > also want a script to drag in the history behind by removing the
> > graft and fetching $BASE^, but that is hard because your repository
> > already "has" that.
> 
> Why not just filter-branch _once at the origin_ and publish the result?

I think the idea was to have a shallow clone starting at a certain
point, as opposed to the --depth option, where you cannot specify a
starting point, but only the depth of the clone.

Björn

Re: Grafts workflow for a "shallow" repository...?

[Michael J Gruber]

Björn Steinbrink venit, vidit, dixit 16.09.2008 10:09:
> On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:
>> "Shawn O. Pearce" <spearce@spearce.org> writes:
>>
>>> Martin Langhoff <martin.langhoff@gmail.com> wrote:
>>>> Here is my attempt at a "let's publish a shallow repository for branch
>>>> of moodle". Let me show you what I did...
>>> ...
>>>>  # 1.7 was a significant release, anything earlier than that
>>>>  # is just not interesting -- even for pickaxe/annotate purposes
>>>>  # so add a graft point right at the branching point.
>>> ...
>>>> Is this kind of workflow (or a variation of it) supported? For this to
>>>> work, we should communicate the grafts in some push operations and
>>>> read them in clone ops - and perhaps in fetch too.
>>> ...
>>> I think that in this case the best thing to do is give users
>>> a shell script that does roughly:
>>>
>>> 	git init
>>> 	echo $BASE >.git/info/grafts
>>> 	git remote add -f origin $url
>>> 	git checkout -b master origin/master
>>>
>>> Sign the script, and let users verify it before executing.  You may
>>> also want a script to drag in the history behind by removing the
>>> graft and fetching $BASE^, but that is hard because your repository
>>> already "has" that.
>> Why not just filter-branch _once at the origin_ and publish the result?
> 
> I think the idea was to have a shallow clone starting at a certain
> point, as opposed to the --depth option, where you cannot specify a
> starting point, but only the depth of the clone.

That's what Junio suggests:

chop the history by introducing an appropriate graft
make it "permanent" by filter-branching (and remove from info/grafts).

Now you have a disconnect dag. clone/push/whatever works on the
"components" given by connected branches.

Note that in this approach all history after the "chopping point" will
be rewritten, i.e. get new sha1's.

Michael

Re: Grafts workflow for a "shallow" repository...?

[Björn Steinbrink]

On 2008.09.16 15:27:43 +0200, Michael J Gruber wrote:
> Björn Steinbrink venit, vidit, dixit 16.09.2008 10:09:
> > On 2008.09.15 23:25:10 -0700, Junio C Hamano wrote:
> >> "Shawn O. Pearce" <spearce@spearce.org> writes:
> >>
> >>> Martin Langhoff <martin.langhoff@gmail.com> wrote:
> >>>> Here is my attempt at a "let's publish a shallow repository for branch
> >>>> of moodle". Let me show you what I did...
> >>> ...
> >>>>  # 1.7 was a significant release, anything earlier than that
> >>>>  # is just not interesting -- even for pickaxe/annotate purposes
> >>>>  # so add a graft point right at the branching point.
> >>> ...
> >>>> Is this kind of workflow (or a variation of it) supported? For this to
> >>>> work, we should communicate the grafts in some push operations and
> >>>> read them in clone ops - and perhaps in fetch too.
> >>> ...
> >>> I think that in this case the best thing to do is give users
> >>> a shell script that does roughly:
> >>>
> >>> 	git init
> >>> 	echo $BASE >.git/info/grafts
> >>> 	git remote add -f origin $url
> >>> 	git checkout -b master origin/master
> >>>
> >>> Sign the script, and let users verify it before executing.  You may
> >>> also want a script to drag in the history behind by removing the
> >>> graft and fetching $BASE^, but that is hard because your repository
> >>> already "has" that.
> >> Why not just filter-branch _once at the origin_ and publish the result?
> > 
> > I think the idea was to have a shallow clone starting at a certain
> > point, as opposed to the --depth option, where you cannot specify a
> > starting point, but only the depth of the clone.
> 
> That's what Junio suggests:
> 
> chop the history by introducing an appropriate graft
> make it "permanent" by filter-branching (and remove from info/grafts).
> 
> Now you have a disconnect dag. clone/push/whatever works on the
> "components" given by connected branches.
> 
> Note that in this approach all history after the "chopping point" will
> be rewritten, i.e. get new sha1's.

But shallow clones using --depth don't get alterted hashed. They have
"graft-like" entries in .git/shallow (or whatever it is called) instead.

Assuming that using a shallow clone is worth the hassle (ie. you have to
fetch significantly less data than with a full clone), I can imagine
some use-cases for that, for example bisecting a bug of which you know
that it was introduced after the last release. Just shallow clone the
repo from that version's tag onwards and bisect locally, while still
being able to use the original hash to report the broken version. Or
genrally anything for which you need a recent part of the history and
which requires you to talk about certain commits. The hashes are useful
for that, and filter-branch'ing breaks that.

Maybe instead of providing "pre-shallowed" repos to clone from, that
should have been an RFE to support shallow clones starting at a given
commit in the DAG? I have no idea whether that's feasible though.

Björn

Re: Grafts workflow for a

[Sergio]

Shawn O. Pearce <spearce <at> spearce.org> writes:

> 
> Martin Langhoff <martin.langhoff <at> gmail.com> wrote:
> > Here is my attempt at a "let's publish a shallow repository for branch
> > of moodle". Let me show you what I did...
> ...
> >  # 1.7 was a significant release, anything earlier than that
> >  # is just not interesting -- even for pickaxe/annotate purposes
> >  # so add a graft point right at the branching point.
> ...
> > Is this kind of workflow (or a variation of it) supported? For this to
> > work, we should communicate the grafts in some push operations and
> > read them in clone ops - and perhaps in fetch too.
> 
> Currently the grafts file isn't transferred over any transport
> protocol as it is considered to be local only to the repository.
> 
> For one thing, grafts are a security risk.  Any user can graft
> anything in at any position and log/blame operations will honor
> the graft, rather than what is stored in the signed commit chain.
> Its a low risk, but it allows a peer to lie to you and give you
> something other than what you asked for.
> 

Would it make sense to differentiate two types of grafts, just like we
differentiate two kind of tags?

Then there could be "annotated grafts objects" that could be optionally signed
and that would get transferred on clones, fetches, pushes, etc.

Sergio

Re: Grafts workflow for a "shallow" repository...?

[Sverre Rabbelier]

On Tue, Sep 16, 2008 at 15:50, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> Maybe instead of providing "pre-shallowed" repos to clone from, that
> should have been an RFE to support shallow clones starting at a given
> commit in the DAG? I have no idea whether that's feasible though.

Would it be like grafts, only the graft is set up by the fetcher,
instead of the host? (E.g., the graft is created on clone, instead of
-before- the clone, by means of the --depth parameter?) Which means
the mentioned security risk is not there (what with the fetcher
setting it up himself).

-- 
Cheers,

Sverre Rabbelier

Re: Grafts workflow for a "shallow" repository...?

[Dmitry Potapov]

On Wed, Sep 17, 2008 at 12:02:25AM +0200, Sverre Rabbelier wrote:
> On Tue, Sep 16, 2008 at 15:50, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> > Maybe instead of providing "pre-shallowed" repos to clone from, that
> > should have been an RFE to support shallow clones starting at a given
> > commit in the DAG? I have no idea whether that's feasible though.
> 
> Would it be like grafts, only the graft is set up by the fetcher,
> instead of the host?

No, the idea is to have it in the $GIT_DIR/shallow format, which differs
from grafts in that it cannot add new parents. So, it can only truncate
history, while grafts adding new parents can modify history in different
ways.

> (E.g., the graft is created on clone, instead of
> -before- the clone, by means of the --depth parameter?) Which means
> the mentioned security risk is not there (what with the fetcher
> setting it up himself).

The important difference between grafts and shallow is that the latter
does not allow you to add new parents. So, you cannot modify history but
only truncate existing one. So I am not sure about what security risk
you are talking about. Obviously, that git blame will attribute all
changes that were done before to the earliest downloaded commit. Other
than that, I don't think that you can fake anything here. All downloaded
history will be correct and that can be verified based on SHA-1.

Also, I believe that the default mode should be to ignore this shallow
file on server unless the user specified the corresponding option to
make a shallow copy.

BTW, the current implementation of shallow copy has some limitations.
See Documentation/technical/shallow.txt for details.

Dmitry