[RFC] deprecating and eventually removing "git relink"?

[RFC] deprecating and eventually removing "git relink"?

[Junio C Hamano]

What do people think about the subject?  As to the timeframe I am thinking
about deprecation at v1.7.9 (late January 2012) and removal at v1.7.12
(early August 2012) [*1*].

The script more-or-less outlived its usefulness in July 2005 when
packfiles were introduced.

You are better off repacking the repositories in the first place before
accumulating so many loose object that linking the same ones between the
repositories would give you major saving.  It is theoretically possible
that two repositories happen to have the same pack and it would give you
saving to hardlink them together, but even in that case, the saving would
not survive repacking unless they are marked with a ".keep" marker, which
the script does not do anyway.

A more useful feature to attack a similar issue would be to make it easier
to use "--reference" aka "objects/info/alternates". Namely:

 (1) devise a way to make it safer by allowing the repository whose
     objects are borrowed by other repositories a way to protect objects
     that it does not need but may be needed by others from repacking; and

 (2) allowing two repositories that started independently to share objects
     using the alternates mechanism after the fact.

but that is a separate issue.


[Footnote]

*1* Both dates were derived from a mechanical "9-week per release cycle".

Re: [RFC] deprecating and eventually removing "git relink"?

[Miles Bader]

Junio C Hamano <gitster@pobox.com> writes:
>  (2) allowing two repositories that started independently to share objects
>      using the alternates mechanism after the fact.

Can they not already?

I mean, it works great right now to do:

  cd $REP2
  echo $REP1/.git/objects > .git/objects/info/alternates
  git gc

Do you mean a more elaborate UI that does this nicely...? or something
else?

It might be nice to have a mechanism where new objects would update
the _alternate_ rather than the object-store in the tree where the
command was run... then you could easily have a bunch of trees using a
central object store without needing to update the central store
occasionally by hand (and do gc in its "clients")...

-Miles

-- 
"Most attacks seem to take place at night, during a rainstorm, uphill,
 where four map sheets join."   -- Anon. British Officer in WW I

Re: [RFC] deprecating and eventually removing "git relink"?

[Junio C Hamano]

Miles Bader <miles@gnu.org> writes:

> Do you mean a more elaborate UI that does this nicely...?

Yes, that is what I meant. I also have a feeling that people would prefer
to have an option that treats these two repositories equally; your
illustration makes one a subordinate to the other.

Re: [RFC] deprecating and eventually removing "git relink"?

[Simon Brenner]

I think one of the most annoying aspects of alternates (beyond the
hassle of adding/removing them except using clone --reference) is the
danger of losing data if you aren't absolutely sure that your
alternate is stable and won't ever lose references to objects.

If the alternate just had links to the referring repositories, I think
this hole could be neatly closed.

On Mon, Nov 14, 2011 at 7:06 AM, Miles Bader <miles@gnu.org> wrote:
> It might be nice to have a mechanism where new objects would update
> the _alternate_ rather than the object-store in the tree where the
> command was run... then you could easily have a bunch of trees using a
> central object store without needing to update the central store
> occasionally by hand (and do gc in its "clients")...

This sounds like a nice way forward: replace/extend the current
alternates system with support for a shared object store that is
"intelligently" shared so that it can be gc:d based on all refs from
all referring repositories. I imagine it would be something very much
like a bare repository - except it wouldn't have any refs of its own,
just a list of other repositories it should search for refs when
GC:ing.

The object store currently built into each git repository could even
become a special case of that: a shared object store (that happens to
reside under .git) with a single referring repository (the parent .git
dir). If the location of the object store is configurable, clone
--reference could simply point the new repository directly to the
shared store instead of ever setting up a local object store.

// Simon

Re: [RFC] deprecating and eventually removing "git relink"?

[Chris Packham]

On 14/11/11 19:27, Junio C Hamano wrote:
> Miles Bader <miles@gnu.org> writes:
> 
>> Do you mean a more elaborate UI that does this nicely...?
> 
> Yes, that is what I meant. I also have a feeling that people would prefer
> to have an option that treats these two repositories equally; your
> illustration makes one a subordinate to the other.

Not sure if it's what you're after but there was this patch [1] that I
was kicking around a while back. I've still got the code in an old
branch if there is interest in resurrecting it. It looks like I started
addressing Junio's comments and never posted v3.

[1] http://article.gmane.org/gmane.comp.version-control.git/143164

Re: [RFC] deprecating and eventually removing "git relink"?

[Junio C Hamano]

Miles Bader <miles@gnu.org> writes:

> It might be nice to have a mechanism where new objects would update
> the _alternate_ rather than the object-store in the tree where the
> command was run.

With the alternate mechanism, your borrowing is read-only and that is
exactly why you can borrow from other peoples' repositories to which you
have no write permission to.

What you are suggesting is fundamentally different from the alternates
mechanism. I am not saying it is better or worse, though. Not yet at this
point in this message.

> .. then you could easily have a bunch of trees using a
> central object store without needing to update the central store
> occasionally by hand (and do gc in its "clients")...

If you write objects to the central store, "gc" in the "clients" will be a
no-op because they do not have their own objects. But instead, crufts your
"clients" accumulate will be in the central store. There is still need for
"gc" at the central store to remove things that are no longer used by any
client, isn't it? Unless you declare that you do not care because perhaps
the central store is large enough, that is.

At least with the alternates, running "gc" in the "clients" is a safe
operation and the only change necessary is to make fsck/repack aware of
the repositories that borrow from the repository these commands are run,
and the logic to do so is exactly the same as the case to run "gc" in your
central store, I would think.

Re: [RFC] deprecating and eventually removing "git relink"?

[Jeff King]

On Mon, Nov 14, 2011 at 09:48:07AM +0100, Simon Brenner wrote:

> On Mon, Nov 14, 2011 at 7:06 AM, Miles Bader <miles@gnu.org> wrote:
> > It might be nice to have a mechanism where new objects would update
> > the _alternate_ rather than the object-store in the tree where the
> > command was run... then you could easily have a bunch of trees using a
> > central object store without needing to update the central store
> > occasionally by hand (and do gc in its "clients")...
> 
> This sounds like a nice way forward: replace/extend the current
> alternates system with support for a shared object store that is
> "intelligently" shared so that it can be gc:d based on all refs from
> all referring repositories. I imagine it would be something very much
> like a bare repository - except it wouldn't have any refs of its own,
> just a list of other repositories it should search for refs when
> GC:ing.

Yes, I think that is sensible. I'm not sure there is even any core git
code to be written. I think a wrapper that does the following would
probably work:

  1. Make new repo groups. E.g.:

       $ git share init foo

     which would be implemented something like:

       ROOT=$HOME/.git-share
       git init --bare $ROOT/$1

  2. Add a repo to a group.

       $ git share add foo

     implemented as:

       echo $ROOT/$1/objects >>.git/objects/info/alternates
       git --git-dir=$ROOT/$1 config --add share.child $PWD

  3. Compact a group.

       $ git share compact foo

     implemented as:

       # delete any existing refs
       git for-each-ref --format='%(refname)' | xargs git update-ref -d

       # now make new refs for each child
       n=1
       for dir in `git config --all share.child`; do
              if ! test -d $dir; then
                      echo >&2 "warning: $dir went away"
                      continue
              fi
              git fetch $dir refs/*:refs/$1/*
              n=$(($n + 1))
       done

       # and then repack/prune
       git repack -ad

       # and then gc each child, dropping anything in the share
       for dir in `git config --all share.child`; do
              git --git-dir=$dir gc
       done

I'm sure I'm missing a corner case or two, and of course there are
quoting issues and error handling missing. But the point is, I don't
think there's a real reason that the UI can't wrap the existing
mechanism, creating a momentary list of refs and pruning based on that.

One issue with this scheme (or most similar schemes) is that child repos
are uniquely identified by their directory name. In the absence of
alternates, it's perfectly reasonable to do:

  git init; hack hack hack; commit commit commit
  cd .. ; mv project new-project-name

but here it would break the shared repo's link to the child (which is
not just inconvenient, but dangerous, as we will not respect its refs
when pruning). Probably the "warning" above should actually error out
and force the user to say "yes, I deleted this child" or "no, I moved it
here".

You could try to be clever with assigning each child a UUID, but then
you have to resort to grepping the filesystem for the UUID to detect a
move. Which is complex and still not foolproof (i.e., if you don't find
it, is it because the repo was deleted, or because it got moved
somewhere that we didn't look?).

-Peff

Re: [RFC] deprecating and eventually removing "git relink"?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, Nov 14, 2011 at 09:48:07AM +0100, Simon Brenner wrote:
>
>> On Mon, Nov 14, 2011 at 7:06 AM, Miles Bader <miles@gnu.org> wrote:
>> > It might be nice to have a mechanism where new objects would update
>> > the _alternate_ rather than the object-store in the tree where the
>> > command was run... then you could easily have a bunch of trees using a
>> > central object store without needing to update the central store
>> > occasionally by hand (and do gc in its "clients")...
>> 
>> This sounds like a nice way forward: replace/extend the current
>> alternates system ...
>
> Yes, I think that is sensible. I'm not sure there is even any core git
> code to be written. I think a wrapper that does the following would
> probably work:

I agree with your outline, which I find is in line with what I had in mind
in the message Miles responded.

The approach is different from what Miles alluded to, which is to have
"clients" create objects in the "central" place in the first place,
though.

Re: [RFC] deprecating and eventually removing "git relink"?

[Jeff King]

On Mon, Nov 14, 2011 at 12:18:25PM -0800, Junio C Hamano wrote:

> > Yes, I think that is sensible. I'm not sure there is even any core git
> > code to be written. I think a wrapper that does the following would
> > probably work:
> 
> I agree with your outline, which I find is in line with what I had in mind
> in the message Miles responded.
> 
> The approach is different from what Miles alluded to, which is to have
> "clients" create objects in the "central" place in the first place,
> though.

It seems to me that is simply an optimization that can come later. An
initial, no-C-code implementation would write to individual repos as
usual, and then occasionally migrate objects to the master shared repo
(and remove duplicates from individual repos). That's an easy to
implement low-risk experiment from which we can draw conclusions about
how well such a system works in practice.

And then if it seems like a good path, an obvious optimization[1] is to
write directly into the parent object store, skipping the migration.
This might involve git-core code, or maybe it just means setting up the
repos differently (e.g., symlinking the objects directory to the master
store).

-Peff

[1] Actually, I am slightly dubious that this optimization is worth
doing. It seems like it would save you from writing the data only to
copy it later. But in practice, we write loose objects, and you are
already rewriting the data to migrate it into packfiles. So the
migration already happens, and instead we would just be migrating to
packfiles in the central repo.

Re: [RFC] deprecating and eventually removing "git relink"?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, Nov 14, 2011 at 12:18:25PM -0800, Junio C Hamano wrote:
>
>> > Yes, I think that is sensible. I'm not sure there is even any core git
>> > code to be written. I think a wrapper that does the following would
>> > probably work:
>> 
>> I agree with your outline, which I find is in line with what I had in mind
>> in the message Miles responded.
>> 
>> The approach is different from what Miles alluded to, which is to have
>> "clients" create objects in the "central" place in the first place,
>> though.
>
> It seems to me that is simply an optimization that can come later.

I did not mean "it is wrong because it does not match what Miles said" by
that. In fact, I think it is a better approach to put things in clients
first and consolidating possible duplicates at the central one purely as
optimization, and I do not necessarily see "write to central from the
beginning" as a particularly good "optimization".

Re: [RFC] deprecating and eventually removing "git relink"?

[Miles Bader]

Junio C Hamano <gitster@pobox.com> writes:
>> It might be nice to have a mechanism where new objects would update
>> the _alternate_ rather than the object-store in the tree where the
>> command was run.
>
> With the alternate mechanism, your borrowing is read-only and that is
> exactly why you can borrow from other peoples' repositories to which you
> have no write permission to.
>
> What you are suggesting is fundamentally different from the alternates
> mechanism. I am not saying it is better or worse, though. Not yet at this
> point in this message.

Sure, and I don't even claim it's a viable idea, just something that
"seems useful."

>> .. then you could easily have a bunch of trees using a central
>> object store without needing to update the central store
>> occasionally by hand (and do gc in its "clients")...
>
> If you write objects to the central store, "gc" in the "clients"
> will be a no-op because they do not have their own objects. But
> instead, crufts your "clients" accumulate will be in the central
> store. There is still need for "gc" at the central store to remove
> things that are no longer used by any client, isn't it? Unless you
> declare that you do not care because perhaps the central store is
> large enough, that is.

Sure, if git had this mode of operation, it would seem desirable for
"git gc" to act on the central store just at the same points it acts
on the "local store" today.

As obviously a gc needs to know all the roots, that suggests the
central store needs to have a list of clients it can scan for roots.

[I suppose the other "problem" is locking; I guess that would
technically be no different that multiple git commands running
simulataneously in the same tree today, but maybe the presence of a
central store would make such situations occur more frequently...]

> At least with the alternates, running "gc" in the "clients" is a
> safe operation and the only change necessary is to make fsck/repack
> aware of the repositories that borrow from the repository these
> commands are run, and the logic to do so is exactly the same as the
> case to run "gc" in your central store, I would think.

Hmmm sure.

-miles

-- 
=====
(^o^;
(()))
*This is the cute octopus virus, please copy it into your sig so it can spread.

Re: [RFC] deprecating and eventually removing "git relink"?

[Miles Bader]

Junio C Hamano <gitster@pobox.com> writes:
> I did not mean "it is wrong because it does not match what Miles said"
> by that. In fact, I think it is a better approach to put things in
> clients first and consolidating possible duplicates at the central one
> purely as optimization, and I do not necessarily see "write to central
> from the beginning" as a particularly good "optimization".

FWIW, this seems reasonable to me...

-Miles

-- 
Circus, n. A place where horses, ponies and elephants are permitted to see
men, women and children acting the fool.

Re: [RFC] deprecating and eventually removing "git relink"?

[Phillip Susi]

On 11/14/2011 5:34 AM, Jeff King wrote:
> One issue with this scheme (or most similar schemes) is that child repos
> are uniquely identified by their directory name. In the absence of
> alternates, it's perfectly reasonable to do:
>
>    git init; hack hack hack; commit commit commit
>    cd .. ; mv project new-project-name
>
> but here it would break the shared repo's link to the child (which is
> not just inconvenient, but dangerous, as we will not respect its refs
> when pruning). Probably the "warning" above should actually error out
> and force the user to say "yes, I deleted this child" or "no, I moved it
> here".

I hacked together a setup a few weeks ago that doesn't suffer from that 
problem.  I had two repos that had considerable shared history ( one 
forked from the other ), so I created a temporary repository and pointed 
its alternates to the other two.  I then did some shell magic to 
generate a list of all objects shared by both repos, and sent that list 
to git-pack-objects.  This gave me a pack file in the temp repo that 
contained all of the shared objects.  I then made a .keep file and hard 
linked this pack file ( and index, and .keep file ) into both original 
repos, deleted the temp repo, and then repacked both original repos. 
This left them both with two pack files: one that is shared, and one 
that is all of the objects specific to that repo.

Because the shared objects are in a pack file that both repos hard link 
to, neither one will break if I (re)move the other.  It would be nice if 
git relink could be enhanced to do this, then you can just periodically 
run relink with a list of repos and it could hard link all of the shared 
data into a big shared pack file, with no need to have a "master" repo 
that requires special handling.

Re: [RFC] deprecating and eventually removing "git relink"?

[Jeff King]

On Mon, Nov 21, 2011 at 05:09:23PM -0500, Phillip Susi wrote:

> I hacked together a setup a few weeks ago that doesn't suffer from
> that problem.  I had two repos that had considerable shared history (
> one forked from the other ), so I created a temporary repository and
> pointed its alternates to the other two.  I then did some shell magic
> to generate a list of all objects shared by both repos, and sent that
> list to git-pack-objects.  This gave me a pack file in the temp repo
> that contained all of the shared objects.  I then made a .keep file
> and hard linked this pack file ( and index, and .keep file ) into
> both original repos, deleted the temp repo, and then repacked both
> original repos. This left them both with two pack files: one that is
> shared, and one that is all of the objects specific to that repo.
> 
> Because the shared objects are in a pack file that both repos hard
> link to, neither one will break if I (re)move the other.

Yes, that is one way to do it. The big drawback there is that by using
hard links, you can only share objects between repos within the same
filesystem.

I think the presence of the '.keep' files should make "git gc" do the
right thing, and not waste space. The relinking procedure is a little
more complex, but that's not a big deal. It's just a periodic
maintenance thing that will happen inside a script (and you would want
to do the periodic maintenance as often as you would with the shared
repo approach).

Nothing is maintaining the list of "here are all of the related repos
that are sharing objects".  Which is a feature in some ways, because you
don't have to care if repos go away or move. But when your periodic "git
relink" comes around, the burden is on the user to redecide the set of
related repos.

So unlike with the shared repo, where "git gc" in a child repo could say
"Oh, I have a shared parent; I should go there and do the parent-gc
there", relinking would be a more manual thing. On the other hand,
nothing is stopping you from building something more automated around
this relink-repos-together building block.

So yeah, I think it's a perfectly reasonable approach, if you don't mind
the hard link requirement, and your relink is something like "git relink
~/linux-repos/*".

Patches? :)

-Peff

Re: [RFC] deprecating and eventually removing "git relink"?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/21/2011 05:19 PM, Jeff King wrote:
> Yes, that is one way to do it. The big drawback there is that by
> using hard links, you can only share objects between repos within
> the same filesystem.

Yep, hard links requires same filesystem, but means you don't have to
have a central repo that you have to gc very carefully.

> So yeah, I think it's a perfectly reasonable approach, if you don't
> mind the hard link requirement, and your relink is something like
> "git relink ~/linux-repos/*".

That's the idea.

To sum up, it appears there are 3 possible implementations:

1) hard link + master repo with mutual awareness
2) hard link + no master repo or inter-repo awareness
3) alternatives + master repo with mutual awareness

With #1 you can auto relink when any child does gc

With #2 the repos don't need to be aware of each other

With #3 the repos don't need to be on the same fs, can auto relink
when any child does gc, but moving a child or removing the master repo
causes breakage

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk7LAcgACgkQJ4UciIs+XuKzrQCeIb2Tb3D+nqDlF5bBD8vkQy/t
4sQAniEbL2kZK2wvY+y4tvd+QDRh1G85
=QHQ5
-----END PGP SIGNATURE-----