[PATCH] [PATCH] git-mirror - exactly mirror another repository

[PATCH] [PATCH] git-mirror - exactly mirror another repository

[Petr Baudis]

Sometimes its handy to be able to efficiently backup or mirror one
Git repository to another Git repository by employing the native
Git object transfer protocol.  But when mirroring or backing up a
repository you really want:

  1) Every object in the source to go to the mirror.
  2) Every ref in the source to go to the mirror.
  3) Any ref removed from the source to be removed from the mirror.

and since git-fetch doesn't do 2 and 3, here's a tool that does.

This is based on Shawn Pearce's patch from 25 Sep 2006, updated to take
Junio's and Sergey's review into account, to use few newer pieces of Git
infrastructure and with few trivial tweaks. The repacking part was dropped
since git-fetch does that on its own now.

I actually still would kind of prefer this to be a git-fetch's feature but
the general mood seems to be to have this as a separate command and I can't
say I care at all.

Signed-off-by: Petr Baudis <pasky@suse.cz>
---

 .gitignore                   |    1 
 Documentation/config.txt     |    6 ++
 Documentation/git-mirror.txt |   54 +++++++++++++++++++++
 Makefile                     |    2 -
 git-mirror.perl              |  110 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 172 insertions(+), 1 deletions(-)

diff --git a/.gitignore b/.gitignore
index 4dc0c39..d0b67da 100644
--- a/.gitignore
+++ b/.gitignore
@@ -80,6 +80,7 @@ git-merge-resolve
 git-merge-stupid
 git-merge-subtree
 git-mergetool
+git-mirror
 git-mktag
 git-mktree
 git-name-rev
diff --git a/Documentation/config.txt b/Documentation/config.txt
index e0aff53..e05e4c5 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -513,6 +513,12 @@ log.showroot::
 	Tools like gitlink:git-log[1] or gitlink:git-whatchanged[1], which
 	normally hide the root commit will now show it. True by default.
 
+mirror.allowed::
+	If true, gitlink:git-mirror[1] will be allowed to run on the
+	repository.  Please see its documentation for all the implications.
+
+mirror.
+
 merge.summary::
 	Whether to include summaries of merged commits in newly created
 	merge commit messages. False by default.
diff --git a/Documentation/git-mirror.txt b/Documentation/git-mirror.txt
new file mode 100644
index 0000000..0909124
--- /dev/null
+++ b/Documentation/git-mirror.txt
@@ -0,0 +1,54 @@
+git-mirror(1)
+============
+
+NAME
+----
+git-mirror - Exactly mirror another repository
+
+
+SYNOPSIS
+--------
+'git-mirror' <repository>
+
+
+DESCRIPTION
+-----------
+Completely mirrors another repository into the local repository.
+
+All heads and tags from the other repository are copied to the
+local repository without any regard for merging.  This means
+that all heads and tags will be FORCIBLY CHANGED in the local
+repository to make them match the other repository.  Any local
+ref or tags which has been deleted from the other repository
+will also be deleted from the local repository.
+
+After mirroring is complete, in case the 'HEAD' symref does not
+seem to match locally the remote one, `git-mirror` will attempt
+to guess which head is now the 'HEAD'; if there are more candidates,
+one will be chosen roughly randomly.
+
+
+CONFIGURATION
+-------------
+
+Prior to updating the local repository 'git-mirror' requires
+that the user set 'mirror.allowed' to a true value in the local
+repository's config file.  This is considered to be a safety
+latch which is intended to prevent accidental overwriting of
+the local repository.
+
+
+SEE ALSO
+--------
+gitlink:git-fetch[1]
+gitlink:git-prune[1]
+gitlink:git-repack[1]
+
+
+Author
+------
+Written by Shawn Pearce <spearce@spearce.org> and Petr Baudis <pasky@suse.cz>
+
+GIT
+---
+Part of the gitlink:git[7] suite
diff --git a/Makefile b/Makefile
index 60c41fd..5fa370f 100644
--- a/Makefile
+++ b/Makefile
@@ -208,7 +208,7 @@ SCRIPT_PERL = \
 	git-archimport.perl git-cvsimport.perl git-relink.perl \
 	git-cvsserver.perl git-remote.perl \
 	git-svnimport.perl git-cvsexportcommit.perl \
-	git-send-email.perl git-svn.perl
+	git-send-email.perl git-svn.perl git-mirror.perl
 
 SCRIPT_PYTHON = \
 	git-p4import.py
diff --git a/git-mirror.perl b/git-mirror.perl
new file mode 100644
index 0000000..22ea879
--- /dev/null
+++ b/git-mirror.perl
@@ -0,0 +1,110 @@
+#!/usr/bin/env perl
+# Copyright (C) 2006, Shawn Pearce <spearce@spearce.org>
+# This file is licensed under the GPL v2, or a later version
+# at the discretion of Linus.
+
+use warnings;
+use strict;
+use Git;
+
+my $remote = shift || 'origin';
+my $repo = Git->repository();
+
+# Verify its OK to execute in this repository.
+#
+unless ($repo->config_bool('mirror.allowed')) {
+	print STDERR <<EOF;
+Error: mirror.allowed is false.
+Error:
+Error: For safety reasons please set mirror.allowed in this repository's
+Error: config before using this command.
+Error:
+Error: Unless you are using this repository ONLY for mirroring another
+Error: repository you probably don't want to do this. Please see the
+Error: manpage for more details.
+EOF
+	exit 1;
+}
+
+# Build our list of refs.
+#
+my $remote_refs = ls_refs($repo, $remote);
+my $local_refs = ls_refs($repo, $repo->repo_path());
+my $remote_HEAD = $remote_refs->{'HEAD'};
+my $local_HEAD = $local_refs->{'HEAD'};
+delete $remote_refs->{'HEAD'};
+delete $local_refs->{'HEAD'};
+
+# Execute the fetch for any refs which differ from our own.
+# We don't worry about trying to optimize for rewinds or
+# exact branch copies as they are rather uncommon.
+#
+my @to_fetch;
+while (my ($ref, $hash) = each %$remote_refs) {
+	push(@to_fetch, "$ref:$ref")
+		if (!$local_refs->{$ref} || $local_refs->{$ref} ne $hash);
+}
+if (@to_fetch) {
+	git_cmd_try {
+		$repo->command_noisy('fetch',
+			'--force',
+			'--update-head-ok',
+			$remote, sort @to_fetch);
+	} '%s failed w/ code %d';
+} else {
+	print "No changed refs.  Skipping fetch.\n";
+}
+
+# See what the remote has HEAD pointing at and update our local
+# HEAD to point at some ref which points at the same hash.
+# Prefer to keep HEAD the same if possible to avoid HEAD drifting
+# between different branches.
+# Note that with dumb protocols, we don't get to *know* HEAD implicitly
+# with git-ls-remote...
+#
+git_cmd_try {
+	my $headref = $repo->command_oneline('symbolic-ref', 'HEAD');
+	my $HEAD;
+	if (not $remote_refs->{$headref}) {
+		$HEAD = 'refs/heads/master';
+		print "Local HEAD branch disappeared, falling back to refs/heads/master\n";
+	} elsif ($remote_HEAD and $remote_refs->{$headref} ne $remote_HEAD) {
+		my %by_hash = map {$remote_refs->{$_} => $_}
+			grep {m,^refs/heads/,}
+			sort keys %$remote_refs;
+		$HEAD = $by_hash{$remote_HEAD};
+		if ($HEAD) {
+			print "Setting HEAD to $HEAD ($remote_HEAD)\n";
+		} else {
+			print "Remote HEAD ($remote_HEAD) does not match any remote branch\n";
+		}
+	}
+	if ($HEAD) {
+		$repo->command_noisy('symbolic-ref', 'HEAD', $HEAD);
+	}
+} '%s failed w/ code %d';
+
+# Delete any local refs which the server no longer contains.
+#
+foreach my $ref (keys %$local_refs) {
+	next if $remote_refs->{$ref};
+	print "Removing $ref\n";
+	git_cmd_try {
+		$repo->command_noisy('update-ref', '-d', $ref, $local_refs->{$ref});
+	} '%s failed w/ code %d';
+}
+
+sub ls_refs {
+	my $repo = shift;
+	my $name = shift;
+	my $refs = $repo->remote_refs($name);
+	my @interesting = grep {
+		$_ eq 'HEAD' or (/^refs\// and not /\.\./ and not /\^{}$/);
+	} keys %$refs;
+
+	my %refs2;
+	# This funky-looking expression puts @interesting-subset of %$refs
+	# to %refs2.
+	@refs2{@interesting} = @{$refs}{@interesting};
+	\%refs2;
+}

Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository

[Petr Baudis]

  Sorry for the dupe mail, accidentally slipped out. Full patch series
show arrive when vger's greylisting permits.

				Petr "Pasky" Baudis

Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository

[A Large Angry SCM]

Petr Baudis wrote:
> Sometimes its handy to be able to efficiently backup or mirror one
> Git repository to another Git repository by employing the native
> Git object transfer protocol.  But when mirroring or backing up a
> repository you really want:
> 
>   1) Every object in the source to go to the mirror.
>   2) Every ref in the source to go to the mirror.
>   3) Any ref removed from the source to be removed from the mirror.
> 
> and since git-fetch doesn't do 2 and 3, here's a tool that does.
> 
> This is based on Shawn Pearce's patch from 25 Sep 2006, updated to take
> Junio's and Sergey's review into account, to use few newer pieces of Git
> infrastructure and with few trivial tweaks. The repacking part was dropped
> since git-fetch does that on its own now.
> 
> I actually still would kind of prefer this to be a git-fetch's feature but
> the general mood seems to be to have this as a separate command and I can't
> say I care at all.
> 

It would be nice if this had an option to not mirror HEAD and another 
option to map the refs from (/ref/[^/])/(.*) to /ref/\1/${repo_path}/\2.

Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository

[Petr Baudis]

On Fri, Apr 27, 2007 at 04:47:07AM CEST, A Large Angry SCM wrote:
> It would be nice if this had an option to not mirror HEAD and another 
> option to map the refs from (/ref/[^/])/(.*) to /ref/\1/${repo_path}/\2.

When I try to guess why you want this, I think this is just saying "it
would be nice if this was part of git-fetch and if git-fetch actually
did this automagically by default". Because when you clone a given
repository with all its branches, you are naturally interested not only
in the snapshot of the branch set at the point of the clone but more
likely just want "anytime whatever branches are over there". So when
fetching a * refspec it's natural to look for new branches and fetch
them too; and if you do that, it should be pretty easy to also (probably
optionally) remove stale branches.

But heck, maybe with all the nifty fetch--tools git-fetch is already
doing it. But there seems to be basically zero documentation, the code
is not that easy to follow and it's too late for me to be decyphering it
now... 

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
		-- Samuel Beckett

Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository

[Johannes Schindelin]

Hi,

On Fri, 27 Apr 2007, Petr Baudis wrote:

> I actually still would kind of prefer this to be a git-fetch's feature 
> but the general mood seems to be to have this as a separate command and 
> I can't say I care at all.

I have to admit that I share your preference. And I really would like to 
be able to say "git fetch --all <remote>", because I use Git for backup 
operations, too.

Ciao,
Dscho

Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository

[Andy Whitcroft]

Petr Baudis wrote:
> Sometimes its handy to be able to efficiently backup or mirror one
> Git repository to another Git repository by employing the native
> Git object transfer protocol.  But when mirroring or backing up a
> repository you really want:
> 
>   1) Every object in the source to go to the mirror.
>   2) Every ref in the source to go to the mirror.
>   3) Any ref removed from the source to be removed from the mirror.
> 
> and since git-fetch doesn't do 2 and 3, here's a tool that does.
> 
> This is based on Shawn Pearce's patch from 25 Sep 2006, updated to take
> Junio's and Sergey's review into account, to use few newer pieces of Git
> infrastructure and with few trivial tweaks. The repacking part was dropped
> since git-fetch does that on its own now.
> 
> I actually still would kind of prefer this to be a git-fetch's feature but
> the general mood seems to be to have this as a separate command and I can't
> say I care at all.
> 
> Signed-off-by: Petr Baudis <pasky@suse.cz>
> ---
> 
>  .gitignore                   |    1 
>  Documentation/config.txt     |    6 ++
>  Documentation/git-mirror.txt |   54 +++++++++++++++++++++
>  Makefile                     |    2 -
>  git-mirror.perl              |  110 ++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 172 insertions(+), 1 deletions(-)
> 
> diff --git a/.gitignore b/.gitignore
> index 4dc0c39..d0b67da 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -80,6 +80,7 @@ git-merge-resolve
>  git-merge-stupid
>  git-merge-subtree
>  git-mergetool
> +git-mirror
>  git-mktag
>  git-mktree
>  git-name-rev
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index e0aff53..e05e4c5 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -513,6 +513,12 @@ log.showroot::
>  	Tools like gitlink:git-log[1] or gitlink:git-whatchanged[1], which
>  	normally hide the root commit will now show it. True by default.
>  
> +mirror.allowed::
> +	If true, gitlink:git-mirror[1] will be allowed to run on the
> +	repository.  Please see its documentation for all the implications.
> +
> +mirror.
> +

Ok, I can see why you'd want to protect a user from going git-mirror
<foo> and losing their local changes.  But you've only protected them
from mirroring at all.  If they want to use this repo as a mirror of
somewhere X you have not stopped them from git-mirror Y'ing themselves.
 Now yes if you know its a mirror then I guess you can recover with a
git-mirror X but ...

It seems to me if you are having a config option that perhaps it should
be the URL from whence the mirror comes, and the command should just be
'git mirror'.

[mirror]
	url='somewhere'

-apw

Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository

[Petr Baudis]

On Fri, Apr 27, 2007 at 06:38:12AM CEST, Petr Baudis wrote:
> But heck, maybe with all the nifty fetch--tools git-fetch is already
> doing it. But there seems to be basically zero documentation, the code
> is not that easy to follow and it's too late for me to be decyphering it
> now... 

Ok, so it seems that git-fetch already can do almost all of this; at
least it seems that I'm far from being alone in not realizing it. :-)

With the refspec +refs/heads/*:refs/heads/* and git remote prune
(thanks, Shawn!), it seems that git-fetch can already do everything
git-mirror tries to do, except updating the HEAD, but especially with
the symref embedding in git-ls-remote that Junio proposed, even that
should be fixable easily, right?

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
		-- Samuel Beckett