[PATCH] Added --mirror-all to git-fetch.

[PATCH] Added --mirror-all to git-fetch.

[Shawn Pearce]

The --mirror-all option to git-fetch can be used to obtain a copy of
every available remote ref into the current repository.  This can be
a rather destructive update as the local repository will have its
HEAD ref overwritten, as well as any ref which it shares in common
with the remote repository.  On the other hand it can be useful if
all you want to do is have this repository track another repository,
such as if you are providing Git repository hosting and mirroring
source repositories on other systems.

Currently local refs are not deleted even if they do not exist in the
remote repository.  This may be taken as either a feature or a bug.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 pasky was asking about this on #git.  So here it is.

 Documentation/fetch-options.txt |    7 ++++++
 git-fetch.sh                    |   43 ++++++++++++++++++++++++++++++++++++---
 2 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 13f34d3..5ed9b4f 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -30,6 +30,13 @@
 	flag lets all tags and their associated objects be
 	downloaded.
 
+\--mirror-all::
+	All refs and tags on the remote side are downloaded to
+	the local side.  This option is not intendend for user
+	repositories as it will overwrite every local ref,
+	including HEAD.  You probably don't mean to use this
+	option.
+
 -k, \--keep::
 	Keep downloaded pack.
 
diff --git a/git-fetch.sh b/git-fetch.sh
index 09a5d6c..d22c560 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -14,6 +14,7 @@ IFS="$LF"
 rloga=fetch
 no_tags=
 tags=
+mirror_all=
 append=
 force=
 verbose=
@@ -36,6 +37,9 @@ do
 	-f|--f|--fo|--for|--forc|--force)
 		force=t
 		;;
+	--mirror-all)
+		mirror_all=t
+		;;
 	-t|--t|--ta|--tag|--tags)
 		tags=t
 		;;
@@ -216,12 +220,45 @@ case "$update_head_ok" in
 	;;
 esac
 
-# If --tags (and later --heads or --all) is specified, then we are
-# not talking about defaults stored in Pull: line of remotes or
+# If --tags or -mirror_all (and later --heads) is specified, then we
+# are not talking about defaults stored in Pull: line of remotes or
 # branches file, and just fetch those and refspecs explicitly given.
 # Otherwise we do what we always did.
 
-reflist=$(get_remote_refs_for_fetch "$@")
+if test "$mirror_all"
+then
+	tags=0;  # tags are implied in all 
+	reflist=`IFS="	" &&
+		  (
+			git-ls-remote $upload_pack "$remote" ||
+			echo fail ouch
+		  ) |
+	          while read sha1 name
+		  do
+			case "$sha1" in
+			fail)
+				exit 1
+			esac
+			case "$name" in
+			*^*) continue ;;
+			esac
+		  	if git-check-ref-format "$name"
+			then
+			    echo ".+${name}:${name}"
+			else
+				if test "x$name" = xHEAD
+				then
+					echo ".+${name}:${name}"
+				else
+					echo >&2 "warning: malformed ref ${name} ignored"
+				fi
+			fi
+		  done` || exit
+else
+	# Not using all; do what we always have done.
+	reflist=$(get_remote_refs_for_fetch "$@")
+fi
+
 if test "$tags"
 then
 	taglist=`IFS="	" &&
-- 
1.4.2.1.g832e

Re: [PATCH] Added --mirror-all to git-fetch.

[Petr Baudis]

Dear diary, on Wed, Sep 20, 2006 at 01:28:51AM CEST, I got a letter
where Shawn Pearce <spearce@spearce.org> said that...
> Currently local refs are not deleted even if they do not exist in the
> remote repository.  This may be taken as either a feature or a bug.

  It's a bug for my purposes, but a minor one.

>  pasky was asking about this on #git.  So here it is.

  Thanks!

> @@ -216,12 +220,45 @@ case "$update_head_ok" in
>  	;;
>  esac
>  
> -# If --tags (and later --heads or --all) is specified, then we are
> -# not talking about defaults stored in Pull: line of remotes or
> +# If --tags or -mirror_all (and later --heads) is specified, then we
                  ^^ missing dash ;-)
> +# are not talking about defaults stored in Pull: line of remotes or
>  # branches file, and just fetch those and refspecs explicitly given.
>  # Otherwise we do what we always did.
>  
> -reflist=$(get_remote_refs_for_fetch "$@")
> +if test "$mirror_all"
> +then
> +	tags=0;  # tags are implied in all 

  This should be tags=;

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam

Re: [PATCH] Added --mirror-all to git-fetch.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> The --mirror-all option to git-fetch can be used to obtain a copy of
> every available remote ref into the current repository.  This can be
> a rather destructive update as the local repository will have its
> HEAD ref overwritten, as well as any ref which it shares in common
> with the remote repository.

I can sort of see where something like this is very much useful,
but it sounds like a tool quite different from git-fetch.

 (1) I really do not like rolling this kind of speciale purpose
     command into git-fetch, which is frequently used by end
     users, to avoid mistakes.

     If there is reluctance against adding yet another new
     command (and there certainly is), this feels more like a
     cousin of "git-clone --bare".

 (2) Although there is no inherent reason not allowing a working
     tree associated with the repository that is kept updated
     this way, the user will be utterly confused in a working
     tree whose current branch head is updated like this, until
     the working tree and the index is matched to the updated
     HEAD.  It might be reasonable to run checkout -f HEAD when
     a working tree is associated with the repository (the
     command is screwing over even the current branch HEAD, so
     losing what happened to be in the working tree is really
     not an issue), but as a much easier safety measure we might
     want to allow this mode of updating only on a bare
     repository (that is, .git/index should not exist).

 (3) This feels primarily meant for something like Pasky is
     trying to run --- mirrored distribution point of git
     repositories perhaps displayed with gitweb.  When updating
     such a repository, you would want to do things like running
     update-server-info and automatically repacking the object
     store.  Especially the latter would be an interesting topic
     (the archive vs active repacking strategy we talked about,
     combined with set of packs with staggered spans to help
     commit walkers Pasky talked about quite a while ago).


> ... 
> such as if you are providing Git repository hosting and mirroring
> source repositories on other systems.
>
> Currently local refs are not deleted even if they do not exist in the
> remote repository.  This may be taken as either a feature or a bug.

For that purpose I would say that is definitely a bug.

Re: [PATCH] Added --mirror-all to git-fetch.

[Petr Baudis]

Dear diary, on Wed, Sep 20, 2006 at 06:06:11PM CEST, I got a letter
where Junio C Hamano <junkio@cox.net> said that...
..snip..
>      If there is reluctance against adding yet another new
>      command (and there certainly is), this feels more like a
>      cousin of "git-clone --bare".

Certainly. But, what exactly are you proposing here? :-) (Besides
possible change of the switch name.) Making this a git-clone option
sounds even much worse.

..snip..
>  (2) Although there is no inherent reason not allowing a working
>      tree associated with the repository that is kept updated
>      this way, the user will be utterly confused in a working
>      tree whose current branch head is updated like this, until
>      the working tree and the index is matched to the updated
>      HEAD.

git-fetch --mirror-all can still be very useful even with a working copy
if you are keeping all the remote heads in .git/refs/remotes/. I'd
venture to say that in that case, this is frequently what you actually
want when fetching from the repository (given that you have already let
git clone do that once).

..snip..
>      (the archive vs active repacking strategy we talked about,

Hmm, I think I've missed this, I must look that in the archive.

>      combined with set of packs with staggered spans to help
>      commit walkers Pasky talked about quite a while ago).

Yes, this is certainly one of things I would like to implement at
repo.or.cz.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam

Re: [PATCH] Added --mirror-all to git-fetch.

[Shawn Pearce]

Petr Baudis <pasky@suse.cz> wrote:
> Dear diary, on Wed, Sep 20, 2006 at 06:06:11PM CEST, I got a letter
> where Junio C Hamano <junkio@cox.net> said that...
[snip]
> >  (2) Although there is no inherent reason not allowing a working
> >      tree associated with the repository that is kept updated
> >      this way, the user will be utterly confused in a working
> >      tree whose current branch head is updated like this, until
> >      the working tree and the index is matched to the updated
> >      HEAD.
> 
> git-fetch --mirror-all can still be very useful even with a working copy
> if you are keeping all the remote heads in .git/refs/remotes/. I'd
> venture to say that in that case, this is frequently what you actually
> want when fetching from the repository (given that you have already let
> git clone do that once).

I think that's more `git fetch --all`.  You are pulling every remote
head available at a given remote into a subdirectory of your own ref
space.  That's rather different than replacing your entire ref space
with the one available on the remote, which is what --mirror-all
is doing and what you wanted for the hosting you are offering.

> ..snip..
> >      (the archive vs active repacking strategy we talked about,
> 
> Hmm, I think I've missed this, I must look that in the archive.

Junio pushed the core code out but nobody has done the Porecelain
for it.  The basic idea is to prevent repacking every pack all of
the time; there's probably no reason to repack a 100 MiB pack file
every time you repack your loose objects so you might want to keep
say a <5 MiB "active pack" holding your recent created objects
and repack that frequently and a larger 100+ MiB "history pack"
holding everything else.  Maybe you repack everything on a longer
time scale, such as once a year.

> >      combined with set of packs with staggered spans to help
> >      commit walkers Pasky talked about quite a while ago).
> 
> Yes, this is certainly one of things I would like to implement at
> repo.or.cz.

Borrowing your line:
Hmm, I think I've missed this, I must look that in the archive.

:-)

-- 
Shawn.

Re: [PATCH] Added --mirror-all to git-fetch.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> Petr Baudis <pasky@suse.cz> wrote:
>> Dear diary, on Wed, Sep 20, 2006 at 06:06:11PM CEST, I got a letter
>> where Junio C Hamano <junkio@cox.net> said that...
> [snip]
>> >  (2) Although there is no inherent reason not allowing a working
>> >      tree associated with the repository that is kept updated
>> >      this way, the user will be utterly confused in a working
>> >      tree whose current branch head is updated like this, until
>> >      the working tree and the index is matched to the updated
>> >      HEAD.
>> 
>> git-fetch --mirror-all can still be very useful even with a working copy
>> if you are keeping all the remote heads in .git/refs/remotes/. I'd
>> venture to say that in that case, this is frequently what you actually
>> want when fetching from the repository (given that you have already let
>> git clone do that once).
>
> I think that's more `git fetch --all`.  You are pulling every remote
> head available at a given remote into a subdirectory of your own ref
> space.  That's rather different than replacing your entire ref space
> with the one available on the remote, which is what --mirror-all
> is doing and what you wanted for the hosting you are offering.

I realize I am going around in circles, but Pasky's "remotes/"
argument made me realize that this mirroring is not much more
than "fetch --force --all".  I initially had an impression that
this was for only strict mirroring where you do not even want
your own "origin", but if you arrange the .git/remotes/origin
file the right way, "fetch --force --all" (if you remembered to
put '+' in front of the refspecs, even without --force) would
what --mirror-all would do wouldn't it?

Having said that, I am not sure if "fetch --all" should delete
branches that do not exist on the remote end.  For mirroring, I
definitely would want the tool to delete them, and that is one
of the reason I feel this is much more similar to git-clone than
git-fetch, unless it is a separate command (perhaps git-mirror).

Re: [PATCH] Added --mirror-all to git-fetch.

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> I realize I am going around in circles, but Pasky's "remotes/"
> argument made me realize that this mirroring is not much more
> than "fetch --force --all".  I initially had an impression that
> this was for only strict mirroring where you do not even want
> your own "origin", but if you arrange the .git/remotes/origin
> file the right way, "fetch --force --all" (if you remembered to
> put '+' in front of the refspecs, even without --force) would
> what --mirror-all would do wouldn't it?

I started this change with '--all' and realized that ideally you
want '--all' to copy all available refs/heads/* from the remote to
refs/remotes/<name>/* here.  You want to create any new branches
which the remote has introduced since your last fetch.

You probably don't want to force a non-fast forward unless there's a
'+' in the corresponding Pull line of remotes/<name> or if --force is
used.  However you probably also want to delete any removed branches.

Which I think is quite different from a mirror.  A mirror wants to
replace the entire ref namespace with what's on the remote as it
has no need for a local namespace of its own.


Originally I gave Pasky a one-liner on #git:

  git fetch --force origin $(git ls-remote origin \
      | awk '{if(!/\^{}$/){print $2":"$2}}')

but he expressed interest in it being a native feature of the
core-Git fetch Porcelain.  To be honest I disagreed with him but
submitted the patch anyway.

I think --all copying into .git/refs/remotes/<name>/* makes perfect
sense.

And I think this mirror thing may make more sense as a small wrapper
around git-fetch.  A wrapper that checks for:

	- its running in a bare repository;
	- it has a single remote named origin;
	- HEAD isn't a symlink or a symref (its a normal ref in its
	  own right);
	- git-mirror.permitted is true in the config file.
 
-- 
Shawn.

Re: [PATCH] Added --mirror-all to git-fetch.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> I started this change with '--all' and realized that ideally you
> want '--all' to copy all available refs/heads/* from the remote to
> refs/remotes/<name>/* here.  You want to create any new branches
> which the remote has introduced since your last fetch.
>
> You probably don't want to force a non-fast forward unless there's a
> '+' in the corresponding Pull line of remotes/<name> or if --force is
> used.  However you probably also want to delete any removed branches.

I think we are in agreement here.

> Which I think is quite different from a mirror.  A mirror wants to
> replace the entire ref namespace with what's on the remote as it
> has no need for a local namespace of its own.

If you have remotes/origin that does not use the refs/remotes
hierarchy, does not copy master->origin (iow, the assumption
that "origin" branch is to be merged into local "master") but
copies remote master to local master, and forces all refs by
default, then your local ref namespace is effectively a mirror
of the remote, so it is very tempting to think --mirror is a
logical extension of "fetch --all".

Except that new refs are not created and old refs are not
deleted.  And these two are rather big exceptions.

If we are to have another command (git-mirror which is a thin
wrapper around git-fetch as you propose later), these two
exceptions would be cleanly handled there.  Otherwise, we would
need to give the command a hint (git-mirror.permitted) to allow
them to be done in addition to the usual "fetch --all", but then
we would end up doing the equivalent of your one-liner anyway,
so there is not much point having it in git-fetch.  So I am in
favor of your thin-wrapper idea.

> Originally I gave Pasky a one-liner on #git:
>
>   git fetch --force origin $(git ls-remote origin \
>       | awk '{if(!/\^{}$/){print $2":"$2}}')

Which sounds a very sane thing to do, especially given git is
designed to be scripted this way.

> And I think this mirror thing may make more sense as a small wrapper
> around git-fetch.  A wrapper that checks for:
>
> 	- its running in a bare repository;

Ok.

> 	- it has a single remote named origin;

Do you mean ".git/remotes must have only one file 'origin' and
nothing else", or do you mean "There must be .git/remotes/foobla
for 'git-mirror foobla' to work"?

> 	- HEAD isn't a symlink or a symref (its a normal ref in its
> 	  own right);

Why?  Usually HEAD is a pointer to the active branch.  I
consider it is a bug in the upload-pack protocol not to convey
that information (but that is a separate topic).

> 	- git-mirror.permitted is true in the config file.

Ok.

Re: [PATCH] Added --mirror-all to git-fetch.

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> > Originally I gave Pasky a one-liner on #git:
> >
> >   git fetch --force origin $(git ls-remote origin \
> >       | awk '{if(!/\^{}$/){print $2":"$2}}')
> 
> Which sounds a very sane thing to do, especially given git is
> designed to be scripted this way.

Exactly.  So I'm not sure why he was against it.  After all he must
have some script that's looping over all mirrored repositories and
invoking git fetch in them.  :-)
 
> > 	- it has a single remote named origin;
> 
> Do you mean ".git/remotes must have only one file 'origin' and
> nothing else", or do you mean "There must be .git/remotes/foobla
> for 'git-mirror foobla' to work"?

I meant ".git/remotes must have only one file 'origin' and
nothing else".  Because what does it mean to mirror two different
repositories at the same time into the same ref namespace?  The most
recently fetched would always win.

But now that I'm writing this I can see someone wanting multiple
remotes so they could store different URLs for the same repository
and call fetch on another if the first isn't responding.  So I
retract this entirely.  So there doesn't even need to be remotes
file; the URL could just be passed on the command line.  A remotes
file (like the default "origin" one) is just a handy place to store
the URL.
 
> > 	- HEAD isn't a symlink or a symref (its a normal ref in its
> > 	  own right);
> 
> Why?  Usually HEAD is a pointer to the active branch.  I
> consider it is a bug in the upload-pack protocol not to convey
> that information (but that is a separate topic).

Because we don't want to update HEAD with the SHA1 that's on the
remote if that's actually going to update some other ref which
doesn't have the same value.  :-)

I think its simple enough to verify HEAD isn't a symlink or symref
(or just doesn't exist) early in the mirror script.  If the user
really wants to do a mirror and they just did a `git init-db --bare`
they could also do a `rm HEAD` before starting `git-mirror`.

But I agree with you entirely about HEAD being a SHA1 and not its
actual link target in the network protocol.  An annoying bug.

-- 
Shawn.

Re: [PATCH] Added --mirror-all to git-fetch.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> I meant ".git/remotes must have only one file 'origin' and
> nothing else".  Because what does it mean to mirror two different
> repositories at the same time into the same ref namespace?  The most
> recently fetched would always win.
>
> But now that I'm writing this I can see someone wanting multiple
> remotes so they could store different URLs for the same repository
> and call fetch on another if the first isn't responding.  So I
> retract this entirely.  So there doesn't even need to be remotes
> file; the URL could just be passed on the command line.  A remotes
> file (like the default "origin" one) is just a handy place to store
> the URL.

Another possibility.  The mirror might want to do something
like:

        .git/refs/remotes/davem/sparc-2.6.git/...
	.git/refs/remotes/torvalds/linux-2.6.git/...
	.git/refs/remotes/jgarzik/netdev-2.6.git/...

to be a super-mirror of related projects.  This would give a
convenient way for the visitors to browse them all at one place,
and would save object store.  There are minor details such as
gitweb may not allow browsing refs/remotes and git-fetch may not
let you fetch from refs/remotes (I haven't checked), but they
are fixable as needed.

> Because we don't want to update HEAD with the SHA1 that's on the
> remote if that's actually going to update some other ref which
> doesn't have the same value.  :-)

What I thought you were going to do with HEAD would be to update
it not by the object name but modifying where it points at.
IOW, if HEAD is a symlink pointing at refs/heads/master and the
remote HEAD now points at refs/heads/next, you do not:

	echo $the_commit_object_name_of_remote_next >.git/HEAD 

which obviously is insane, but you would do

	rm -f .git/HEAD && ln -s refs/heads/next .git/HEAD

> But I agree with you entirely about HEAD being a SHA1 and not its
> actual link target in the network protocol.  An annoying bug.

Which fortunately is fixable ;-).

Re: [PATCH] Added --mirror-all to git-fetch.

[Petr Baudis]

Dear diary, on Wed, Sep 20, 2006 at 06:21:45PM CEST, I got a letter
where Shawn Pearce <spearce@spearce.org> said that...
> > >      combined with set of packs with staggered spans to help
> > >      commit walkers Pasky talked about quite a while ago).
> > 
> > Yes, this is certainly one of things I would like to implement at
> > repo.or.cz.
> 
> Borrowing your line:
> Hmm, I think I've missed this, I must look that in the archive.
> :-)

I've proposed something long time ago in
<20051112135947.GC30496@pasky.or.cz>.

Dear diary, on Wed, Sep 20, 2006 at 07:31:26PM CEST, I got a letter
where Shawn Pearce <spearce@spearce.org> said that...
> Junio C Hamano <junkio@cox.net> wrote:
> > Shawn Pearce <spearce@spearce.org> writes:
> > > Originally I gave Pasky a one-liner on #git:
> > >
> > >   git fetch --force origin $(git ls-remote origin \
> > >       | awk '{if(!/\^{}$/){print $2":"$2}}')
> > 
> > Which sounds a very sane thing to do, especially given git is
> > designed to be scripted this way.
> 
> Exactly.  So I'm not sure why he was against it.  After all he must
> have some script that's looping over all mirrored repositories and
> invoking git fetch in them.  :-)

Basically, I imagined that what I wanted would be very close or
identical to git fetch --all (or whatever you call it, just doing it
with .git/refs/remotes) which I think should certainly be supported
without funny awk pipes. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
lK[d2%Sa2/d0$^Ixp"|dc`;s/\W//g;$_=pack('H*',/((..)*)$/)

Re: [PATCH] Added --mirror-all to git-fetch.

[A Large Angry SCM]

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]

Junio C Hamano wrote:
[...]
> 
> Another possibility.  The mirror might want to do something
> like:
> 
>         .git/refs/remotes/davem/sparc-2.6.git/...
> 	.git/refs/remotes/torvalds/linux-2.6.git/...
> 	.git/refs/remotes/jgarzik/netdev-2.6.git/...
> 
> to be a super-mirror of related projects.  This would give a
> convenient way for the visitors to browse them all at one place,
> and would save object store.  There are minor details such as
> gitweb may not allow browsing refs/remotes and git-fetch may not
> let you fetch from refs/remotes (I haven't checked), but they
> are fixable as needed.
> 

This is what I do with the 2 attached, poorly named, scripts. Very 
convenient for watching some of my favorite projects. ;-)

[-- Attachment #2: fred.sh --]
[-- Type: application/x-shellscript, Size: 432 bytes --]

[-- Attachment #3: fred_history --]
[-- Type: text/plain, Size: 1188 bytes --]

export GIT_COMMITTER_NAME="@"
export GIT_AUTHOR_NAME="@"

./fred.sh git://www.kernel.org/pub/scm/git/git.git git/git

./fred.sh git://www.kernel.org/pub/scm/gitk/gitk.git gitk/gitk

./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.11.y.git stable/linux-2.6.11.y
./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.12.y.git stable/linux-2.6.12.y
./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.13.y.git stable/linux-2.6.13.y
./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.14.y.git stable/linux-2.6.14.y
./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.15.y.git stable/linux-2.6.15.y
./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.16.y.git stable/linux-2.6.16.y
./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.17.y.git stable/linux-2.6.17.y

./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git tglx/history

./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git torvalds/linux-2.6
./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/torvalds/sparse.git torvalds/sparse

Re: [PATCH] Added --mirror-all to git-fetch.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

>> >      (the archive vs active repacking strategy we talked about,
>> 
>> Hmm, I think I've missed this, I must look that in the archive.
>
> Junio pushed the core code out but nobody has done the Porecelain
> for it.  The basic idea is to prevent repacking every pack all of
> the time; there's probably no reason to repack a 100 MiB pack file
> every time you repack your loose objects so you might want to keep
> say a <5 MiB "active pack" holding your recent created objects
> and repack that frequently and a larger 100+ MiB "history pack"
> holding everything else.  Maybe you repack everything on a longer
> time scale, such as once a year.

And IIRC we were stuck on the convention to tell which ones are
archives and which ones are actives (i.e. fair game for
repacking).  We were trying to figure out how to mark active
ones (whose name changes every time you repack).

I realized that there is a very simple and obvious solution for
it.  Introduce $GIT_OBJECT_DIRECTORY/info/archived-packs, a flat
text file that lists the names of the archive packs.  Any pack
not listed there are active one and are subject to repacking.

Re: [PATCH] Added --mirror-all to git-fetch.

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> 
> >> >      (the archive vs active repacking strategy we talked about,
> >> 
> >> Hmm, I think I've missed this, I must look that in the archive.
> >
> > Junio pushed the core code out but nobody has done the Porecelain
> > for it.  The basic idea is to prevent repacking every pack all of
> > the time; there's probably no reason to repack a 100 MiB pack file
> > every time you repack your loose objects so you might want to keep
> > say a <5 MiB "active pack" holding your recent created objects
> > and repack that frequently and a larger 100+ MiB "history pack"
> > holding everything else.  Maybe you repack everything on a longer
> > time scale, such as once a year.
> 
> And IIRC we were stuck on the convention to tell which ones are
> archives and which ones are actives (i.e. fair game for
> repacking).  We were trying to figure out how to mark active
> ones (whose name changes every time you repack).

Yes.  I had pretty much given up on a size threshold concept and
was thus in agreement with you that we just needed a way to flag
each pack file...
 
> I realized that there is a very simple and obvious solution for
> it.  Introduce $GIT_OBJECT_DIRECTORY/info/archived-packs, a flat
> text file that lists the names of the archive packs.  Any pack
> not listed there are active one and are subject to repacking.

Like that.  :-)

-- 
Shawn.