How to stop sharing objects between repositories

How to stop sharing objects between repositories

[Jon Jensen]

Hello.

Situation: I used "git clone -s" to share objects between repositories. 
That creates .git/objects/info/alternates, which points to the other 
repository. Later I need to remove the dependency and make the dependent 
repository self-sufficient, i.e. it should have all the objects internal 
to itself.

I've looked around for a way to do that but haven't found either tools or 
instructions.

I came up with this manual way, copying over the unique remote objects and 
then removing the alternate repository pointer:

(cd /path/to/alternate/repo.git/objects && tar cp .) | (cd .git/objects && tar xvpk)
# some objects will already exist and be skipped, leading to an error on exit, which is fine
rm .git/objects/info/alternates
# or if there's more than one and you're only removing one, edit the alternates file and remove only that pointer

(With GNU tar -C the copy is a little simpler.)

I posted this to the wiki:

http://git.or.cz/gitwiki/GitFaq#Howtostopsharingobjectsbetweenrepositories.3F

If there's a better or built-in way to do this with Git tools, I'd like to 
learn it, and I'd be happy to update the wiki accordingly.

Thanks,
Jon

-- 
Jon Jensen
End Point Corporation
http://www.endpoint.com/

Re: How to stop sharing objects between repositories

[Johannes Schindelin]

Hi,

On Sat, 15 Aug 2009, Jon Jensen wrote:

> If there's a better or built-in way to do this with Git tools, I'd like 
> to learn it, and I'd be happy to update the wiki accordingly.

I think what you need is done by

	git repack -l

(I agree it is not well documented, and I'd welcome a documentation 
patch.)

Ciao,
Dscho

Re: How to stop sharing objects between repositories

[Jeff King]

On Sun, Aug 16, 2009 at 10:43:11AM +0200, Johannes Schindelin wrote:

> > If there's a better or built-in way to do this with Git tools, I'd like 
> > to learn it, and I'd be happy to update the wiki accordingly.
> 
> I think what you need is done by
> 
> 	git repack -l
> 
> (I agree it is not well documented, and I'd welcome a documentation 
> patch.)

I think it is the opposite; packing _without_ "-l" will create a pack
with objects from the alternate; using "-l" suppresses them. Running
"git repack -a" should do the trick, I believe (and you need the "-a" to
ensure that objects already packed in the repo are re-packed).

-Peff

Re: How to stop sharing objects between repositories

[Johannes Schindelin]

Hi,

On Sun, 16 Aug 2009, Jeff King wrote:

> On Sun, Aug 16, 2009 at 10:43:11AM +0200, Johannes Schindelin wrote:
> 
> > > If there's a better or built-in way to do this with Git tools, I'd like 
> > > to learn it, and I'd be happy to update the wiki accordingly.
> > 
> > I think what you need is done by
> > 
> > 	git repack -l
> > 
> > (I agree it is not well documented, and I'd welcome a documentation 
> > patch.)
> 
> I think it is the opposite; packing _without_ "-l" will create a pack
> with objects from the alternate; using "-l" suppresses them. Running
> "git repack -a" should do the trick, I believe (and you need the "-a" to
> ensure that objects already packed in the repo are re-packed).

Hmm.  I really would like a documentation patch, then.

Ciao,
Dscho

Re: How to stop sharing objects between repositories

[Daniel Villeneuve]

Johannes Schindelin wrote:
> Hmm.  I really would like a documentation patch, then.
>
>   
As another way to do it, I've used something along the lines from
    http://article.gmane.org/gmane.comp.version-control.git/62062
namely:

<script>
gitdir=$(git rev-parse --git-dir)
[ -n "$gitdir" ] || die "cannot find Git directory"

cd "$gitdir"
a=objects/info/alternates
if [ -f $a ]; then
  git rev-parse --all HEAD | git pack-objects --revs objects/pack/pack
  rm $a
fi
</script>

I was not sure HEAD would be included via --all (e.g. HEAD pointing to a 
dangling commit), so I added it explicitly.

The reverse operation (enabling sharing for a standalone repository) is 
described here
    
http://git.or.cz/gitwiki/GitFaq#Howtoshareobjectsbetweenexistingrepositories.3F

--
Daniel

Re: How to stop sharing objects between repositories

[Johannes Schindelin]

Hi,

On Sun, 16 Aug 2009, Daniel Villeneuve wrote:

> Johannes Schindelin wrote:
>
> > Hmm.  I really would like a documentation patch, then.
>
> As another way to do it, I've used something along the lines from
>    http://article.gmane.org/gmane.comp.version-control.git/62062

What does this have to do with my comment?

Besides, you are teaching general Git users how to use plumbing.  That is 
asking for pain, and the pain will come back to the Git developers, not to 
you.

Ciao,
Dscho

Re: How to stop sharing objects between repositories

[Jeff King]

On Sun, Aug 16, 2009 at 02:30:15PM +0200, Johannes Schindelin wrote:

> > I think it is the opposite; packing _without_ "-l" will create a pack
> > with objects from the alternate; using "-l" suppresses them. Running
> > "git repack -a" should do the trick, I believe (and you need the "-a" to
> > ensure that objects already packed in the repo are re-packed).
> 
> Hmm.  I really would like a documentation patch, then.

To what? From git-repack(1):

  -l
     Pass the --local option to git-pack-objects. See git-pack-objects(1).

>From git-pack-objects(1):

  --local
      This flag is similar to --incremental; instead of ignoring all
      packed objects, it only ignores objects that are packed and/or not
      in the local object store (i.e. borrowed from an alternate).

So I think the documentation is correct, but for the original poster, it
suffers from two problems:

  1. The impact of "-l" in this case is a bit subtle and confusing.
     I.e., it is not an obvious "use this flag to break the dependency
     on an alternate", but rather "don't use this flag so that you won't
     ignore non-local objects when packing". In fact, you don't need to
     know about it at all

  2. He has to know that "git repack" is the right place to look in the
     first place (_and_ he has to figure out that "-l" is what he cares
     about and follow the docs to git-pack-objects to find out what it
     does).

So I think the best thing is a "by the way, here is how you break this
dependency" closer to where the concept of alternates is defined. I
guess the best part would be under the "-s" flag of git-clone, since
that is presumably how such a situation was created (unless the user is
savvy enough to edit .git/objects/info/alternates themselves, in which
case I think we have to assume they know what they are doing).

So maybe something like this would be enough:

-- >8 --
Subject: [PATCH] docs: mention how to break alternates dependency

A user who has created a repository dependency by using "git
clone -s" does not necessarily know where to look to find
out how to break that dependency. Let's mention it right
under "-s", where they are most likely to find it.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-clone.txt |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index b14de6c..87fa687 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -72,6 +72,11 @@ These objects may be removed by normal git operations (such as 'git-commit')
 which automatically call `git gc --auto`. (See linkgit:git-gc[1].)
 If these objects are removed and were referenced by the cloned repository,
 then the cloned repository will become corrupt.
++
+To break the dependency of the cloned repository to the source
+repository, run `git repack -a` in the cloned repository, which will
+create a new pack in that repository with all referenced objects,
+including those in the source repository.
 
 
 
-- 
1.6.4.257.gb8ef

Re: How to stop sharing objects between repositories

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Subject: [PATCH] docs: mention how to break alternates dependency
>
> A user who has created a repository dependency by using "git
> clone -s" does not necessarily know where to look to find
> out how to break that dependency. Let's mention it right
> under "-s", where they are most likely to find it.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  Documentation/git-clone.txt |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
>
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index b14de6c..87fa687 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -72,6 +72,11 @@ These objects may be removed by normal git operations (such as 'git-commit')
>  which automatically call `git gc --auto`. (See linkgit:git-gc[1].)
>  If these objects are removed and were referenced by the cloned repository,
>  then the cloned repository will become corrupt.
> ++
> +To break the dependency of the cloned repository to the source
> +repository, run `git repack -a` in the cloned repository, which will
> +create a new pack in that repository with all referenced objects,
> +including those in the source repository.

After reading this, two points come to my mind.  They may or may not be
issues.

 (1) Such a user does not necessarily know a casual "git repack -a" breaks
     the dependency, defeating the -s option s/he deliberately used in
     order to save disk space in the first place.  Perhaps we can reword
     this further to kill two penguins with a single stone?

	Note that the pack resulting from running `git repack -a` in the
	repository cloned with the `-s` option will include objects that
	are borrowed from the source repository.  It essentially breaks
	the dependency created by cloning with the `-s` option by copying
	the objects from the source repository.  To keep borrowing from
	the source repository to save disk space, do not use `repack -a`.

     We should suggest an alternative immediately after this sentence,
     e.g. "Instead, use `repack -l`" or something, but somebody should
     check if it is a valid/viable alternative.

 (2) IIRC, "git gc --auto" runs "repack -A".  What is its effect with
     respect to this dependency between object stores?  I suspect it would
     also break the dependency, but if so, is it a good thing?  Perhaps
     should we change it to use a version that keeps the dependency
     instead?

Re: How to stop sharing objects between repositories

[Mike Galbraith]

On Sun, 2009-08-16 at 12:16 -0700, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
> 
> > Subject: [PATCH] docs: mention how to break alternates dependency
> >
> > A user who has created a repository dependency by using "git
> > clone -s" does not necessarily know where to look to find
> > out how to break that dependency. Let's mention it right
> > under "-s", where they are most likely to find it.
> >
> > Signed-off-by: Jeff King <peff@peff.net>
> > ---
> >  Documentation/git-clone.txt |    5 +++++
> >  1 files changed, 5 insertions(+), 0 deletions(-)
> >
> > diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> > index b14de6c..87fa687 100644
> > --- a/Documentation/git-clone.txt
> > +++ b/Documentation/git-clone.txt
> > @@ -72,6 +72,11 @@ These objects may be removed by normal git operations (such as 'git-commit')
> >  which automatically call `git gc --auto`. (See linkgit:git-gc[1].)
> >  If these objects are removed and were referenced by the cloned repository,
> >  then the cloned repository will become corrupt.
> > ++
> > +To break the dependency of the cloned repository to the source
> > +repository, run `git repack -a` in the cloned repository, which will
> > +create a new pack in that repository with all referenced objects,
> > +including those in the source repository.
> 
> After reading this, two points come to my mind.  They may or may not be
> issues.
> 
>  (1) Such a user does not necessarily know a casual "git repack -a" breaks
>      the dependency, defeating the -s option s/he deliberately used in
>      order to save disk space in the first place.  Perhaps we can reword
>      this further to kill two penguins with a single stone?

Perhaps a runtime warning that you're about to break it?  This user may
not even be the one who set the thing up, no?

	-T. Peanut Gallery

Re: How to stop sharing objects between repositories

[Jeff King]

On Mon, Aug 17, 2009 at 04:21:22AM +0200, Mike Galbraith wrote:

> >  (1) Such a user does not necessarily know a casual "git repack -a" breaks
> >      the dependency, defeating the -s option s/he deliberately used in
> >      order to save disk space in the first place.  Perhaps we can reword
> >      this further to kill two penguins with a single stone?
> 
> Perhaps a runtime warning that you're about to break it?  This user may
> not even be the one who set the thing up, no?

I'm not really sure what such a setup would look like. If it is a big
hosting site like kernel.org or repo.or.cz, then probably it wouldn't
matter much. The admins there should probably be running "git repack -l
-d -A" periodically to consolidate the object stores (which can happen
from this sort of repacking, or from people just pushing the same
commits to their repos).

That being said, I can see there being setups where such a warning might
be useful. However, we don't really know if the user _wants_ that
effect, or if it is an accident. So people following the recommnded
"here is how you break the dependency" advice will also get the warning.

I'm torn on whether this is actually a good idea.

-- >8 --
Subject: [PATCH] repack: warn when "-l" is not used with alternates

Failing to use "-l" means that we will copy objects from the
source repository, nullifying the usefulness of "-s". We
don't want to make this an error, though, since "git repack
-a" is used to intentionally break the dependency.

Signed-off-by: Jeff King <peff@peff.net>
---
Is "test -s" portable? It's in POSIX, but I have some lingering doubt in
the back of my mind.

A user seeing such a warning can perhaps ^C to abort the pack. However,
should we also give instructions on how to undo the copying (which
should be "git repack -d -l -A")?

 git-repack.sh |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/git-repack.sh b/git-repack.sh
index 1eb3bca..0bdc6e9 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -44,6 +44,15 @@ do
 	shift
 done
 
+if test -z "$local" && test -s "$GIT_DIR/objects/info/alternates"; then
+cat >&2 <<'EOF'
+warning: this repository uses objects from other repositories via the
+warning: "alternates" mechanism; repacking without "-l" will cause objects
+warning: to be copied into this repository, wasting disk space.
+
+EOF
+fi
+
 case "`git config --bool repack.usedeltabaseoffset || echo true`" in
 true)
 	extra="$extra --delta-base-offset" ;;
-- 
1.6.4.283.ga2765.dirty

Re: How to stop sharing objects between repositories

[Mike Galbraith]

On Mon, 2009-08-17 at 02:48 -0400, Jeff King wrote:
> On Mon, Aug 17, 2009 at 04:21:22AM +0200, Mike Galbraith wrote:
> 
> > >  (1) Such a user does not necessarily know a casual "git repack -a" breaks
> > >      the dependency, defeating the -s option s/he deliberately used in
> > >      order to save disk space in the first place.  Perhaps we can reword
> > >      this further to kill two penguins with a single stone?
> > 
> > Perhaps a runtime warning that you're about to break it?  This user may
> > not even be the one who set the thing up, no?
> 
> I'm not really sure what such a setup would look like. If it is a big
> hosting site like kernel.org or repo.or.cz, then probably it wouldn't
> matter much. The admins there should probably be running "git repack -l
> -d -A" periodically to consolidate the object stores (which can happen
> from this sort of repacking, or from people just pushing the same
> commits to their repos).
> 
> That being said, I can see there being setups where such a warning might
> be useful. However, we don't really know if the user _wants_ that
> effect, or if it is an accident. So people following the recommnded
> "here is how you break the dependency" advice will also get the warning.
> 
> I'm torn on whether this is actually a good idea.

Yeah.  There are any number of ways to shoot oneself in the foot, and
while on the one hand idiot proofing can be nice if you were about to
screw up, "really really?" messages are most frequently annoying noise.

	-Mike

Re: How to stop sharing objects between repositories

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, Aug 17, 2009 at 04:21:22AM +0200, Mike Galbraith wrote:
>
>> >  (1) Such a user does not necessarily know a casual "git repack -a" breaks
>> >      the dependency, defeating the -s option s/he deliberately used in
>> >      order to save disk space in the first place.  Perhaps we can reword
>> >      this further to kill two penguins with a single stone?
>> 
>> Perhaps a runtime warning that you're about to break it?  This user may
>> not even be the one who set the thing up, no?
>
> I'm not really sure what such a setup would look like....
> ...
> That being said, I can see there being setups where such a warning might
> be useful. However, we don't really know if the user _wants_ that
> effect, or if it is an accident.
> ...
> "here is how you break the dependency" advice will also get the warning.
>
> I'm torn on whether this is actually a good idea.

I would understand if you were torn if the proposed change were to refuse
to run without -l in a repository with alternates when --force is not
given, or something of that nature.

But I can tell you that this "just warn" cannot be a good idea for a very
simple reason: breaking and then warning is useless---it is too late for
the user to do anything about it.

Re: How to stop sharing objects between repositories

[Jeff King]

On Mon, Aug 17, 2009 at 12:24:22AM -0700, Junio C Hamano wrote:

> > I'm torn on whether this is actually a good idea.
> 
> I would understand if you were torn if the proposed change were to refuse
> to run without -l in a repository with alternates when --force is not
> given, or something of that nature.
> 
> But I can tell you that this "just warn" cannot be a good idea for a very
> simple reason: breaking and then warning is useless---it is too late for
> the user to do anything about it.

Did you miss the part where I asked "should we include instructions to
the user on how to fix this"?

-Peff

Re: How to stop sharing objects between repositories

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>> But I can tell you that this "just warn" cannot be a good idea for a very
>> simple reason: breaking and then warning is useless---it is too late for
>> the user to do anything about it.
>
> Did you miss the part where I asked "should we include instructions to
> the user on how to fix this"?

Actually, I didn't.  It is very hard to lose data once you put it in git;
"by following recovery insn the user can redo" is often trivially correct
thanks to it.

But it is not a very good option to cause the damage and then give
recovery insn.  The user might have ran out of quota, and even if he
didn't, he wasted needless cycles for the unwanted sort of repacking.

Re: How to stop sharing objects between repositories

[Jeff King]

On Mon, Aug 17, 2009 at 12:35:53AM -0700, Junio C Hamano wrote:

> > Did you miss the part where I asked "should we include instructions to
> > the user on how to fix this"?
> 
> Actually, I didn't.  It is very hard to lose data once you put it in git;
> "by following recovery insn the user can redo" is often trivially correct
> thanks to it.
> 
> But it is not a very good option to cause the damage and then give
> recovery insn.  The user might have ran out of quota, and even if he
> didn't, he wasted needless cycles for the unwanted sort of repacking.

OK, let's forget the warning, then. Hopefully between the note under
"clone -s" and the fact that most people should be using "git gc" these
days, it won't be a big issue.

-Peff

Re: How to stop sharing objects between repositories

[Jeff King]

On Sun, Aug 16, 2009 at 12:16:22PM -0700, Junio C Hamano wrote:

> After reading this, two points come to my mind.  They may or may not be
> issues.
> 
>  (1) Such a user does not necessarily know a casual "git repack -a" breaks
>      the dependency, defeating the -s option s/he deliberately used in
>      order to save disk space in the first place.  Perhaps we can reword
>      this further to kill two penguins with a single stone?
> 
> 	Note that the pack resulting from running `git repack -a` in the
> 	repository cloned with the `-s` option will include objects that
> 	are borrowed from the source repository.  It essentially breaks
> 	the dependency created by cloning with the `-s` option by copying
> 	the objects from the source repository.  To keep borrowing from
> 	the source repository to save disk space, do not use `repack -a`.

Good point, but I don't think this wording is quite right. You can also
cause such an inefficiency by simply running "git repack", if the source
has loose objects. In other words:

  1. "git repack -a" is sufficient to break dependency, as it copies
     both packed and loose objects

  2. "git repack" _may_ break the dependency, if there are no packs, as
     it copies only loose objects. It _may_ introduce inefficiency, but
     only if there are loose objects.

  3. "git repack -l" always keeps the dependency and current efficiency.

     As an aside, making this list makes me realize there is no easy
     "keep the dependency and increase efficiency". In other words, pack
     everything that is not available otherwise, and then prune the
     remaining packs.

Modified patch is below.

>      We should suggest an alternative immediately after this sentence,
>      e.g. "Instead, use `repack -l`" or something, but somebody should
>      check if it is a valid/viable alternative.

It does work. From a user's perspective, I think "-l" would probably be
a more sane default. But I think it is off for historical reasons, and
these days we try to steer users towards "git gc", anyway, which does
use "-l" by default.

-- >8 --
Subject: [PATCH] docs: describe impact of repack on "clone -s"

The effects of repacking on a repository with alternates are
a bit subtle. The two main things users will want are:

  1. Not to waste disk space by accidentally copying objects
     which could be shared.

  2. Copying all objects explicitly to break the dependency
     on the source repo.

This patch describes both under the "clone -s"
documentation. It makes sense to put it there rather than in
git-repack.txt for both cases. For (1), we are warning the
user who is using "clone -s" about what _not_ to do, so we
need to get their attention when reading about "clone -s".
For (2), we are telling them how git-repack can be used to
accomplish a task, but until they know that git-repack is
the right tool, they have no reason to look at the repack
documentation.

Signed-off-by: Jeff King <peff@peff.net>
---

The extra deleted lines in the patch below are just cleaning up some
excess whitespace.

 Documentation/git-clone.txt |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index b14de6c..b25944f 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -72,8 +72,16 @@ These objects may be removed by normal git operations (such as 'git-commit')
 which automatically call `git gc --auto`. (See linkgit:git-gc[1].)
 If these objects are removed and were referenced by the cloned repository,
 then the cloned repository will become corrupt.
-
-
++
+Note that running `git repack` without the `-l` option in a repository
+cloned with `-s` will copy objects from the source repository into a
+pack in the cloned repository, removing the disk space savings of `clone
+-s`. It is safe, however, to run `git gc`, which uses the `-l` option by
+default.
++
+If you want to break the dependency of a repository cloned with `-s` on
+its source repository, you can simply run `git repack -a` to copy all
+objects from the source repository into a pack in the cloned repository.
 
 --reference <repository>::
 	If the reference repository is on the local machine
-- 
1.6.4.283.gec993

Re: How to stop sharing objects between repositories

[Jeff King]

On Mon, Aug 17, 2009 at 02:19:16AM -0400, Jeff King wrote:

>   1. "git repack -a" is sufficient to break dependency, as it copies
>      both packed and loose objects
> 
>   2. "git repack" _may_ break the dependency, if there are no packs, as
>      it copies only loose objects. It _may_ introduce inefficiency, but
>      only if there are loose objects.
> 
>   3. "git repack -l" always keeps the dependency and current efficiency.
> 
>      As an aside, making this list makes me realize there is no easy
>      "keep the dependency and increase efficiency". In other words, pack
>      everything that is not available otherwise, and then prune the
>      remaining packs.

OK, I take it back. "git repack -d -l -A" will get rid of any packs that
are redundant with your alternate.

-Peff

Re: How to stop sharing objects between repositories

[Jeff King]

On Sun, Aug 16, 2009 at 12:16:22PM -0700, Junio C Hamano wrote:

>  (2) IIRC, "git gc --auto" runs "repack -A".  What is its effect with
>      respect to this dependency between object stores?  I suspect it would
>      also break the dependency, but if so, is it a good thing?  Perhaps
>      should we change it to use a version that keeps the dependency
>      instead?

No, it actually runs "repack -d -l -A", which behaves fine. I even
tested it to make sure.

BTW, the "gc.auto" setting is really annoying at low levels (I set
gc.auto to 1 for testing). For efficiency, it looks at only one
hashed object directory, and then assumes the other 255 contain roughly
the same number of objects. But you get bad sampling error when you have
fewer than 256 objects. I don't think it is worth caring about, though.
It doesn't seem very sane to set gc.auto to something so low.

-Peff