[PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Shawn Pearce]

Sometimes its handy to be able to efficiently backup or mirror one
Git repository to another Git repository by employing the native
Git object transfer protocol.  But when mirroring or backing up a
repository you really want:

  1) Every object in the source to go to the mirror.
  2) Every ref in the source to go to the mirror.
  3) Any ref removed from the source to be removed from the mirror.
  4) Automatically repack and prune the mirror when necessary.

and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 Since 'git fetch --mirror-all' wasn't quite ready here's something
 slightly more polished.

 I'm hoping this gets accepted as I'm thinking it would be very
 useful for backing up a user's repository to another host on a
 regular basis, not to mention also useful for Pasky's mirroring
 service.  I'm planning on using it for backups, especially of
 repositories where branches are frequently being added and deleted.

 .gitignore                   |    1 
 Documentation/git-mirror.txt |   64 ++++++++++++++++++++++++
 Makefile                     |    2 -
 git-mirror.perl              |  111 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 177 insertions(+), 1 deletions(-)

diff --git a/.gitignore b/.gitignore
index 78a3a3d..5f99149 100644
--- a/.gitignore
+++ b/.gitignore
@@ -68,6 +68,7 @@ git-merge-recur
 git-merge-recursive
 git-merge-resolve
 git-merge-stupid
+git-mirror
 git-mktag
 git-mktree
 git-name-rev
diff --git a/Documentation/git-mirror.txt b/Documentation/git-mirror.txt
new file mode 100644
index 0000000..27e5167
--- /dev/null
+++ b/Documentation/git-mirror.txt
@@ -0,0 +1,64 @@
+git-mirror(1)
+============
+
+NAME
+----
+git-mirror - Exactly mirror another repository.
+
+
+SYNOPSIS
+--------
+'git-mirror' <repository>
+
+
+DESCRIPTION
+-----------
+Completely mirrors another repository into the local repository.
+
+All heads and tags from the other repository are copied to the
+local repository without any regard for merging.  This means
+that all heads and tags will be forcibly changed in the local
+repository to make them match the other repository.  Any local
+ref or tags which has been deleted from the other repository
+will also be deleted from the local repository.
+
+After mirroring is complete the 'HEAD' symref will point at
+any branch in 'refs/heads' which has the same SHA1 as the other
+repository's 'HEAD' contained when 'git-mirror' started.  This
+is simply a rough guess and may not always be accurate.
+
+This command will also invoke 'git-repack' and 'git-prune' if
+the number of loose objects exceeds the configured threshold.
+As a result this command may cause the local repository to lose
+commits that have been removed from the other repository.
+
+
+CONFIGURATION
+-------------
+
+Prior to updating the local repository 'git-mirror' requires
+that the user set 'mirror.allowed' to a true value in the local
+repository's config file.  This is considered to be a safety
+feature which is intended to prevent accidental overwriting of
+the local repository.
+
+After updating the local repository 'git-mirror' will run
+'git-repack -a -d' and 'git-prune' if the number of loose objects
+exceeds the limit specified in the configuration file by
+'mirror.maxlooseobjects' (default 100).
+
+
+SEE ALSO
+--------
+gitlink:git-fetch[1]
+gitlink:git-prune[1]
+gitlink:git-repack[1]
+
+
+Author
+------
+Written by Shawn Pearce <spearce@spearce.org>
+
+GIT
+---
+Part of the gitlink:git[7] suite
diff --git a/Makefile b/Makefile
index 9a1f23f..08fb714 100644
--- a/Makefile
+++ b/Makefile
@@ -181,7 +181,7 @@ SCRIPT_PERL = \
 	git-shortlog.perl git-rerere.perl \
 	git-annotate.perl git-cvsserver.perl \
 	git-svnimport.perl git-cvsexportcommit.perl \
-	git-send-email.perl git-svn.perl
+	git-send-email.perl git-svn.perl git-mirror.perl
 
 SCRIPT_PYTHON = \
 	git-merge-recursive.py
diff --git a/git-mirror.perl b/git-mirror.perl
new file mode 100755
index 0000000..bff2003
--- /dev/null
+++ b/git-mirror.perl
@@ -0,0 +1,111 @@
+#!/usr/bin/env perl
+# Copyright (C) 2006, Shawn Pearce <spearce@spearce.org>
+# This file is licensed under the GPL v2, or a later version
+# at the discretion of Linus.
+
+use warnings;
+use strict;
+use Git;
+
+sub ls_refs ($$);
+
+my $remote = shift || 'origin';
+my $repo = Git->repository();
+
+# Verify its OK to execute in this repository.
+#
+my $mirror_ok = $repo->config('mirror.allowed') || 0;
+unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {
+	print STDERR <<EOF;
+error: mirror.allowed is false.
+error:
+error: For safety reasons please set mirror.allowed in this repository's
+error: config before using this command.
+error:
+error: Unless you are using this repository ONLY for mirroring another
+error: repository you probably don't want to do this.
+EOF
+	exit 1;
+}
+
+# Build our list of refs.
+#
+my $remote_refs = ls_refs($repo, $remote);
+my $local_refs = ls_refs($repo, $repo->repo_path());
+my $remote_HEAD = $remote_refs->{'HEAD'};
+delete $remote_refs->{'HEAD'};
+delete $local_refs->{'HEAD'};
+
+# Delete any local refs which the server no longer contains.
+#
+foreach my $ref (keys %$local_refs) {
+	next if $remote_refs->{$ref};
+	print "removing $ref\n";
+	my $log = "logs/$ref";
+	unlink($repo->repo_path() . '/' . $ref);
+	unlink($repo->repo_path() . '/' . $log);
+	rmdir($repo->repo_path() . '/' . $ref) while $ref =~ s,/[^/]*$,,;
+	rmdir($repo->repo_path() . '/' . $log) while $log =~ s,/[^/]*$,,;
+}
+
+# Execute the fetch for any refs which differ from our own.
+# We don't worry about trying to optimize for rewinds or
+# exact branch copies as they are rather uncommon.
+#
+my @to_fetch;
+while (my ($ref, $hash) = each %$remote_refs) {
+	push(@to_fetch, "$ref:$ref")
+		if (!$local_refs->{$ref} || $local_refs->{$ref} ne $hash);
+}
+if (@to_fetch) {
+	git_cmd_try {
+		$repo->command_noisy('fetch',
+			'--force',
+			'--update-head-ok',
+			$remote, sort @to_fetch);
+	} '%s failed w/ code %d';
+} else {
+	print "No changed refs.  Skipping fetch.\n";
+}
+
+# See what the remote has HEAD pointing at and update our local
+# HEAD to point at the any ref which points at the same hash.
+#
+my %by_hash = map {$remote_refs->{$_} => $_}
+	grep {m,^refs/heads/,}
+	keys %$remote_refs;
+my $HEAD = $by_hash{$remote_HEAD} || 'refs/heads/master';
+print "Setting HEAD to $HEAD\n";
+print "                ($remote_HEAD)\n";
+git_cmd_try {
+	$repo->command_noisy('symbolic-ref', 'HEAD', $HEAD);
+} '%s failed w/ code %d';
+
+# Repack if we have a large number of loose objects.
+#
+if (@to_fetch) {
+	my $count_output = $repo->command('count-objects');
+	my ($cur_loose) = ($count_output =~ /^(\d+) objects/);
+	my $max_loose = $repo->config('mirror.maxlooseobjects') || 100;
+	if ($cur_loose >= $max_loose) {
+		git_cmd_try {
+			$repo->command_noisy('repack', '-a', '-d');
+			$repo->command_noisy('prune');
+		} '%s failed w/ code %d';
+	}
+}
+
+sub ls_refs ($$) {
+	my $repo = shift;
+	my $name = shift;
+	my ($fh, $c) = $repo->command_output_pipe('ls-remote', $name);
+	my %refs;
+	while (<$fh>) {
+		chomp;
+		next if /\^{}$/;
+		my ($hash, $ref) = split(/\t/, $_, 2);
+		$refs{$ref} = $hash if ($ref eq 'HEAD' || $ref =~ m,^refs/,);
+	}
+	$repo->command_close_pipe($fh, $c);
+	\%refs;
+}
-- 
1.4.2.1.gde2b2-dirty

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> Sometimes its handy to be able to efficiently backup or mirror one
> Git repository to another Git repository by employing the native
> Git object transfer protocol.  But when mirroring or backing up a
> repository you really want:
>
>   1) Every object in the source to go to the mirror.
>   2) Every ref in the source to go to the mirror.
>   3) Any ref removed from the source to be removed from the mirror.
>   4) Automatically repack and prune the mirror when necessary.
>
> and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.

Just a note.  I usually use git-push the other way for backups,
and I believe that is how Linus does it, too.

> diff --git a/git-mirror.perl b/git-mirror.perl
> new file mode 100755
> index 0000000..bff2003
> --- /dev/null
> +++ b/git-mirror.perl
> @@ -0,0 +1,111 @@
> +#!/usr/bin/env perl

Please don't.  "#!/usr/bin/env perl" is a disease.

> +# This file is licensed under the GPL v2, or a later version
> +# at the discretion of Linus.

Heh ;-).

> +use warnings;
> +use strict;
> +use Git;
> +
> +sub ls_refs ($$);

I wonder why people like line-noise prototypes.  Do you ever
call ls_refs with parameters that benefit from this?  Otherwise
I prefer not to see them.

> +my $remote = shift || 'origin';
> +my $repo = Git->repository();
> +
> +# Verify its OK to execute in this repository.
> +#
> +my $mirror_ok = $repo->config('mirror.allowed') || 0;
> +unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {

This _is_ ugly.  Doesn't $repo->config() know how to drive
underlying "git-repo-config" with specific type argument?

> +# Delete any local refs which the server no longer contains.
> +#
> +foreach my $ref (keys %$local_refs) {
> +	next if $remote_refs->{$ref};
> +	print "removing $ref\n";
> +	my $log = "logs/$ref";
> +	unlink($repo->repo_path() . '/' . $ref);
> +	unlink($repo->repo_path() . '/' . $log);
> +	rmdir($repo->repo_path() . '/' . $ref) while $ref =~ s,/[^/]*$,,;
> +	rmdir($repo->repo_path() . '/' . $log) while $log =~ s,/[^/]*$,,;
> +}

If you do this upfront and then you lose connection while the
real fetch, next fetch may need to take a lot longer than needed
because it cannot rely on the refs you are losing here.  Ref
removal is rather a rare event, so we may not care too much
about it, though.

> +# Execute the fetch for any refs which differ from our own.
> +# We don't worry about trying to optimize for rewinds or
> +# exact branch copies as they are rather uncommon.

If we need to support only git-native protocols, all of this
optimization is not needed at all.  It's kind of sad that we
need to support commit walkers...

> +if (@to_fetch) {
> +	git_cmd_try {
> +		$repo->command_noisy('fetch',
> +			'--force',
> +			'--update-head-ok',
> +			$remote, sort @to_fetch);
> +	} '%s failed w/ code %d';

Why sort (no objection, just curious)?

> +# Repack if we have a large number of loose objects.
> +#
> +if (@to_fetch) {
> +	my $count_output = $repo->command('count-objects');
> +	my ($cur_loose) = ($count_output =~ /^(\d+) objects/);
> +	my $max_loose = $repo->config('mirror.maxlooseobjects') || 100;
> +	if ($cur_loose >= $max_loose) {
> +		git_cmd_try {
> +			$repo->command_noisy('repack', '-a', '-d');
> +			$repo->command_noisy('prune');
> +		} '%s failed w/ code %d';
> +	}
> +}

If we truly have a large number of objects (in pack and loose),
you do not want to do "repack -a -d", do you?

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> 
> > Sometimes its handy to be able to efficiently backup or mirror one
> > Git repository to another Git repository by employing the native
> > Git object transfer protocol.  But when mirroring or backing up a
> > repository you really want:
> >
> >   1) Every object in the source to go to the mirror.
> >   2) Every ref in the source to go to the mirror.
> >   3) Any ref removed from the source to be removed from the mirror.
> >   4) Automatically repack and prune the mirror when necessary.
> >
> > and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.
> 
> Just a note.  I usually use git-push the other way for backups,
> and I believe that is how Linus does it, too.

I do the same thing right now but it doesn't get all topic branches
unless I name them on the push line or put them into my remotes
file, and when I delete a topic branch it doesn't remove it from
the remote during the next push...  :-)
 
> > diff --git a/git-mirror.perl b/git-mirror.perl
> > new file mode 100755
> > index 0000000..bff2003
> > --- /dev/null
> > +++ b/git-mirror.perl
> > @@ -0,0 +1,111 @@
> > +#!/usr/bin/env perl
> 
> Please don't.  "#!/usr/bin/env perl" is a disease.

I entirely blame git-svn.perl.  I copied that line from there.
 
> > +# This file is licensed under the GPL v2, or a later version
> > +# at the discretion of Linus.
> 
> Heh ;-).

Yea, I thought you'd get a kick out of that, especially after the
recent discussion. :)

> > +use warnings;
> > +use strict;
> > +use Git;
> > +
> > +sub ls_refs ($$);
> 
> I wonder why people like line-noise prototypes.  Do you ever
> call ls_refs with parameters that benefit from this?  Otherwise
> I prefer not to see them.

Perl is line noise.  I've gotten into the habit of prototyping
most of my functions but clearly this one could be omitted without
any problems.
 
> > +my $remote = shift || 'origin';
> > +my $repo = Git->repository();
> > +
> > +# Verify its OK to execute in this repository.
> > +#
> > +my $mirror_ok = $repo->config('mirror.allowed') || 0;
> > +unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {
> 
> This _is_ ugly.  Doesn't $repo->config() know how to drive
> underlying "git-repo-config" with specific type argument?

Agreed.  No it doesn't according to Git.pm.  I probably should have
fixed Git.pm first.
 
> > +# Execute the fetch for any refs which differ from our own.
> > +# We don't worry about trying to optimize for rewinds or
> > +# exact branch copies as they are rather uncommon.
> 
> If we need to support only git-native protocols, all of this
> optimization is not needed at all.  It's kind of sad that we
> need to support commit walkers...

Hmm.  I tested this by mirroring a local Git clone ("../git") and
found it was MUCH faster even when only one head was different.
The large number of tags really made it take a lot longer.  And that
was local/local though the native protocols.  I figured it was
worth the few lines of Perl code.
 
> > +if (@to_fetch) {
> > +	git_cmd_try {
> > +		$repo->command_noisy('fetch',
> > +			'--force',
> > +			'--update-head-ok',
> > +			$remote, sort @to_fetch);
> > +	} '%s failed w/ code %d';
> 
> Why sort (no objection, just curious)?

I'm a freak.  I typically don't like things that have a "randomness"
to them.  Since I'm pulling from keys %foo the order I'm getting refs
back in is "unknown" (up to Perl's hash function).  Sorting them
before using them cleans up that randomness.  Although its not
really random here as the hash function is deterministic.

Maybe its because I work with SQL databases all of the time and
you pretty much can't rely on anything coming back in any sort of
order unless you explicitly force it.  So I tend to do the same
with a lot of other systems.

> > +# Repack if we have a large number of loose objects.
> > +#
> > +if (@to_fetch) {
> > +	my $count_output = $repo->command('count-objects');
> > +	my ($cur_loose) = ($count_output =~ /^(\d+) objects/);
> > +	my $max_loose = $repo->config('mirror.maxlooseobjects') || 100;
> > +	if ($cur_loose >= $max_loose) {
> > +		git_cmd_try {
> > +			$repo->command_noisy('repack', '-a', '-d');
> > +			$repo->command_noisy('prune');
> > +		} '%s failed w/ code %d';
> > +	}
> > +}
> 
> If we truly have a large number of objects (in pack and loose),
> you do not want to do "repack -a -d", do you?

Yes.  Because then I want to get your new --unpacked= option
into git-repack.sh.  The -a should then repack all active packs,
omitting the archive packs.  Which are the larger packs that are
costly to repack.

In my opinion mirroring should be a no-brain require activity.
This script is designed for exactly tracking another repository
without any additional intervention from the user.  Carrying a
huge number of loose objects is not ideal, except for maybe the
HTTP commit walker.

-- 
Shawn.

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Sergey Vlasov]

[-- Attachment #1: Type: text/plain, Size: 6212 bytes --]

On Mon, 25 Sep 2006 00:46:41 -0400 Shawn Pearce wrote:

> Sometimes its handy to be able to efficiently backup or mirror one
> Git repository to another Git repository by employing the native
> Git object transfer protocol.  But when mirroring or backing up a
> repository you really want:
>
>   1) Every object in the source to go to the mirror.
>   2) Every ref in the source to go to the mirror.
>   3) Any ref removed from the source to be removed from the mirror.
>   4) Automatically repack and prune the mirror when necessary.
>
> and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.

Is this patch forgotten, abandoned or what?

[...]
> --- /dev/null
> +++ b/git-mirror.perl
> @@ -0,0 +1,111 @@
> +#!/usr/bin/env perl
> +# Copyright (C) 2006, Shawn Pearce <spearce@spearce.org>
> +# This file is licensed under the GPL v2, or a later version
> +# at the discretion of Linus.
> +
> +use warnings;
> +use strict;
> +use Git;
> +
> +sub ls_refs ($$);
> +
> +my $remote = shift || 'origin';
> +my $repo = Git->repository();
> +
> +# Verify its OK to execute in this repository.
> +#
> +my $mirror_ok = $repo->config('mirror.allowed') || 0;
> +unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {
> +	print STDERR <<EOF;
> +error: mirror.allowed is false.
> +error:
> +error: For safety reasons please set mirror.allowed in this repository's
> +error: config before using this command.
> +error:
> +error: Unless you are using this repository ONLY for mirroring another
> +error: repository you probably don't want to do this.
> +EOF
> +	exit 1;
> +}
> +
> +# Build our list of refs.
> +#
> +my $remote_refs = ls_refs($repo, $remote);
> +my $local_refs = ls_refs($repo, $repo->repo_path());
> +my $remote_HEAD = $remote_refs->{'HEAD'};
> +delete $remote_refs->{'HEAD'};
> +delete $local_refs->{'HEAD'};
> +
> +# Delete any local refs which the server no longer contains.
> +#
> +foreach my $ref (keys %$local_refs) {
> +	next if $remote_refs->{$ref};
> +	print "removing $ref\n";
> +	my $log = "logs/$ref";
> +	unlink($repo->repo_path() . '/' . $ref);
> +	unlink($repo->repo_path() . '/' . $log);
> +	rmdir($repo->repo_path() . '/' . $ref) while $ref =~ s,/[^/]*$,,;
> +	rmdir($repo->repo_path() . '/' . $log) while $log =~ s,/[^/]*$,,;

One more instance of direct ref manipulation...

What is the current state of packed-refs implementation?  Would it be
possible to merge at least the interface part (git-update-ref -d for
deleting refs) while keeping the current ref representation, so that
other programs could use proper interfaces now?

> +}

It would be better to remove old refs after the fetch, because if
someone has renamed a branch, and you remove the old ref before the
fetch, you will fetch all commits for this branch again, even if they
were already available under a different name.  However, this may create
problems during git-fetch due to file/directory conflicts.

Another option is to rename deleted refs to something which does not
collide with other refs - git-fetch will see them irrespective of the
name.

Yet another option is to select temporary names for all fetched refs to
be passed to git-fetch, then read their SHA-1 values after git-fetch
(they might differ from the previous result of git-ls-remote), update
normal refs and delete temporary refs.

BTW, I was thinking about the possibility to save removed refs under,
e.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be
saved there - this will ensure that no object will ever disappear from
the mirror, no matter what is done on the master side.  Obviously, in
this case remote refs like refs/old/* should be filtered.

> +
> +# Execute the fetch for any refs which differ from our own.
> +# We don't worry about trying to optimize for rewinds or
> +# exact branch copies as they are rather uncommon.
> +#
> +my @to_fetch;
> +while (my ($ref, $hash) = each %$remote_refs) {
> +	push(@to_fetch, "$ref:$ref")
> +		if (!$local_refs->{$ref} || $local_refs->{$ref} ne $hash);
> +}
> +if (@to_fetch) {
> +	git_cmd_try {
> +		$repo->command_noisy('fetch',
> +			'--force',
> +			'--update-head-ok',
> +			$remote, sort @to_fetch);

This will fail for repositories with large number of refs due to the
limit on command line size.

I'm not sure how to figure out the limit for splitting this to multiple
git-fetch calls.  And using xargs fed from a pipe means that we cannot
use existing Git.pm interfaces.

> +	} '%s failed w/ code %d';
> +} else {
> +	print "No changed refs.  Skipping fetch.\n";
> +}
> +
> +# See what the remote has HEAD pointing at and update our local
> +# HEAD to point at the any ref which points at the same hash.
> +#
> +my %by_hash = map {$remote_refs->{$_} => $_}
> +	grep {m,^refs/heads/,}
> +	keys %$remote_refs;
> +my $HEAD = $by_hash{$remote_HEAD} || 'refs/heads/master';
> +print "Setting HEAD to $HEAD\n";
> +print "                ($remote_HEAD)\n";
> +git_cmd_try {
> +	$repo->command_noisy('symbolic-ref', 'HEAD', $HEAD);
> +} '%s failed w/ code %d';
> +
> +# Repack if we have a large number of loose objects.
> +#
> +if (@to_fetch) {
> +	my $count_output = $repo->command('count-objects');
> +	my ($cur_loose) = ($count_output =~ /^(\d+) objects/);
> +	my $max_loose = $repo->config('mirror.maxlooseobjects') || 100;
> +	if ($cur_loose >= $max_loose) {
> +		git_cmd_try {
> +			$repo->command_noisy('repack', '-a', '-d');
> +			$repo->command_noisy('prune');

git-prune might be unsafe when used concurrently with other
operations...  This is probably not a concern for a mirror repository,
if something ensures that two mirroring processes for the same
repository cannot be started.

> +		} '%s failed w/ code %d';
> +	}
> +}
> +
> +sub ls_refs ($$) {
> +	my $repo = shift;
> +	my $name = shift;
> +	my ($fh, $c) = $repo->command_output_pipe('ls-remote', $name);
> +	my %refs;
> +	while (<$fh>) {
> +		chomp;
> +		next if /\^{}$/;
> +		my ($hash, $ref) = split(/\t/, $_, 2);
> +		$refs{$ref} = $hash if ($ref eq 'HEAD' || $ref =~ m,^refs/,);

Probably more sanity checks would be good here (to make sure that the
other end does not feed you bogus ref names or hashes).

> +	}
> +	$repo->command_close_pipe($fh, $c);
> +	\%refs;
> +}
> --
> 1.4.2.1.gde2b2-dirty
>

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Shawn Pearce]

Sergey Vlasov <vsu@altlinux.ru> wrote:
> On Mon, 25 Sep 2006 00:46:41 -0400 Shawn Pearce wrote:
> 
> > Sometimes its handy to be able to efficiently backup or mirror one
> > Git repository to another Git repository by employing the native
> > Git object transfer protocol.  But when mirroring or backing up a
> > repository you really want:
> >
> >   1) Every object in the source to go to the mirror.
> >   2) Every ref in the source to go to the mirror.
> >   3) Any ref removed from the source to be removed from the mirror.
> >   4) Automatically repack and prune the mirror when necessary.
> >
> > and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.
> 
> Is this patch forgotten, abandoned or what?

Its waiting around for someone to clean it up.  :-)

Junio didn't accept it as there were a number of issues that he
identified in the patch.  I still have them in my inbox but have
not had time to go back and fix them.  Since I haven't actually had
a real need for git-mirror it has been low on my priority list of
"git things to do".

Your message pointed out a number of issues with the current version
that would be worth fixing before accepting it into git.git.  I agree
with many of them, especially about the direct ref manipulation.
git-mirror was written before the Linus packed ref stuff came
along, so git-update-ref didn't have a -d option at the time...
otherwise I would have used it.

-- 

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Junio C Hamano]

Sergey Vlasov <vsu@altlinux.ru> writes:

> What is the current state of packed-refs implementation?

Scheduled for "master" probably tomorrow, with some disclaimers.

> BTW, I was thinking about the possibility to save removed refs under,
> e.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be
> saved there - this will ensure that no object will ever disappear from
> the mirror, no matter what is done on the master side.  Obviously, in
> this case remote refs like refs/old/* should be filtered.

I do not think that belongs to git-mirror.  It might be better to
have that option in update-ref machinery so that you would even
be protected from a `git branch -d` ran by mistake.

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Jakub Narebski]

Junio C Hamano wrote:

> Sergey Vlasov <vsu@altlinux.ru> writes:
>  
>> BTW, I was thinking about the possibility to save removed refs under,
>> e.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be
>> saved there - this will ensure that no object will ever disappear from
>> the mirror, no matter what is done on the master side.  Obviously, in
>> this case remote refs like refs/old/* should be filtered.
> 
> I do not think that belongs to git-mirror.  It might be better to
> have that option in update-ref machinery so that you would even
> be protected from a `git branch -d` ran by mistake.

Perhaps just don't remove log when branch is deleted. I tend for example
to be careful when reordering commits on a branch, and use git branch -f
instead of deleting and recreating branch to not lose reflog.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Sergey Vlasov]

[-- Attachment #1: Type: text/plain, Size: 1892 bytes --]

On Wed, 01 Nov 2006 02:08:08 +0100 Jakub Narebski wrote:

> Junio C Hamano wrote:
> > Sergey Vlasov <vsu@altlinux.ru> writes:
> >> BTW, I was thinking about the possibility to save removed refs under,
> >> e.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be
> >> saved there - this will ensure that no object will ever disappear from
> >> the mirror, no matter what is done on the master side.  Obviously, in
> >> this case remote refs like refs/old/* should be filtered.
> >
> > I do not think that belongs to git-mirror.  It might be better to
> > have that option in update-ref machinery so that you would even
> > be protected from a `git branch -d` ran by mistake.

I agree - this would be even better.

What name format should be used for such saved refs?  refs/old/`date -I`
is not unique enough; probably `date --utc +%Y.%m.%d-%H.%M.%S`?  And it
would be good if multiple refs which were deleted or modified in a
non-fast-forward way during a single operation (like git-mirror) would
be saved together - which may be tricky if they are saved at the lower
level (in update-ref).

Adding the fast-forward check into update-ref also does not look nice,
but this check is required for full safety.

> Perhaps just don't remove log when branch is deleted. I tend for example
> to be careful when reordering commits on a branch, and use git branch -f
> instead of deleting and recreating branch to not lose reflog.

Keeping the reflog does not help - reflog entries are not refs, and do
not keep the referenced objects live in the repository.  Therefore old
objects will be lost when the repository is repacked and pruned.

Also, the reflog really must be deleted together with the ref itself -
otherwise it may cause a file/directory conflict when another ref is
created.  Of course, if the ref is renamed instead of deleting, the
corresponding reflog may be renamed too.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Jakub Narebski]

Sergey Vlasov wrote:

>> Perhaps just don't remove log when branch is deleted. I tend for example
>> to be careful when reordering commits on a branch, and use git branch -f
>> instead of deleting and recreating branch to not lose reflog.
> 
> Keeping the reflog does not help - reflog entries are not refs, and do
> not keep the referenced objects live in the repository.  Therefore old
> objects will be lost when the repository is repacked and pruned.

Well, now to reorder and choose which commits on branch to save, and
optionally also rebase, I use
$ git branch tmp
$ git branch -f <branch> <upstream>
$ git checkout <branch>
$ git cherry-pick tmp~<n>
...
$ git branch -D tmp

> Also, the reflog really must be deleted together with the ref itself -
> otherwise it may cause a file/directory conflict when another ref is
> created.  Of course, if the ref is renamed instead of deleting, the
> corresponding reflog may be renamed too.

There is currently no command to rename branch (and rename reflog).
It would be nice if git-branch could also rename branch (and reflog).
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.

[Junio C Hamano]

Sergey Vlasov <vsu@altlinux.ru> writes:

> What name format should be used for such saved refs?  refs/old/`date -I`
> is not unique enough; probably `date --utc +%Y.%m.%d-%H.%M.%S`?  And it
> would be good if multiple refs which were deleted or modified in a
> non-fast-forward way during a single operation (like git-mirror) would
> be saved together - which may be tricky if they are saved at the lower
> level (in update-ref).
>
> Adding the fast-forward check into update-ref also does not look nice,
> but this check is required for full safety.

I wasn't going to suggest doing the check or even naming at such
a low level.  From the usability point of view, the caller
should decide if the discarded refs are even saved at all, and
if so in which namespace.  That way commit_lock_file() can just
notice the old version of the locked file (it is always sitting
next to it) and hardlink the discarded one to the purgatory
before renaming the newly created .lock file there.