Client-side mirroring patches (v0)

Client-side mirroring patches (v0)

[Sam Vilain]

Hey folks, this is the first stage of git mirroring - making the
client support it in the face of a completely ignorant server.  I
intended to make the clean-up better, but I've been sitting on these
for a couple of weeks so I thought it would be better to have them out
there for people to have a squiz at.

The next stage would be for the mirror list to be communicated to
clients over the network protocol and updated in the git config.

Also there is the matter of falling over to the next mirror should one
not be reachable, but then we're getting into C weaknesses really.
Should I plan to do exception recovery using 'longjmp' ?  Also the
process should be interruptible and provide a user menu.  Again this
seems like it would be very tedious and clumsy in C.  How do people
manage?  Anyway, enjoy... Sam

[PATCH 1/4] remote: allow mirroring to be specified, and document settings

[Sam Vilain]

Add a per-remote setting that can list alternate URLs that the same
repository (or a close fork) can be found, a fetch command-line
switch, a config flag to enable it by default, and finally a method
for selecting a default mirror.

If the preferred mirror does not exist in the list of mirrors it is
considered invalid, in preparation for the time when the mirror list
can be supplied or added to by the upload-pack protocol response.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 Documentation/config.txt        |   13 +++++++++++++
 Documentation/fetch-options.txt |   24 ++++++++++++++++++++++++
 builtin-fetch.c                 |    4 +++-
 remote.c                        |   17 +++++++++++++++++
 remote.h                        |    6 ++++++
 5 files changed, 63 insertions(+), 1 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index d1e2120..edde0e4 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1375,6 +1375,19 @@ remote.<name>.url::
 remote.<name>.pushurl::
 	The push URL of a remote repository.  See linkgit:git-push[1].
 
+remote.<name>.mirror-url::
+	An alternate URL which should have many similar refs to the
+	real remote at least most of the time.  This option can be
+	specified multiple times.  See linkgit:git-fetch[1].
+
+remote.<name>.use-mirror::
+	Prefer to contact mirrors first (boolean).  See
+	linkgit:git-fetch[1].
+
+remote.<name>.preferred-mirror::
+	Specify which mirror to try first (full URL).  May be updated
+	by user interaction during 'fetch'.  See linkgit:git-fetch[1].
+
 remote.<name>.proxy::
 	For remotes that require curl (http, https and ftp), the URL to
 	the proxy to use for that remote.  Set to the empty string to
diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 2886874..8f07a56 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -62,6 +62,30 @@ ifndef::git-pull[]
 	Pass --quiet to git-fetch-pack and silence any other internally
 	used git commands.
 
+-M::
+--use-mirror::
+	Try to use a configured mirror for the bulk of the transfer
+	rather than the main upstream.  See gitlink:git-config[1] for
+	the appropriate options to set.
++
+The preferred mirror (or the first if no preferred URL is defined) is
+first contacted and fetched from, and any refs it presents which match
+the source <refspec> (including the rules for fetching tags) are saved
+under `refs/mirrors/`<remote>`/hostname`.  If there is a network
+timeout, or the user interrupts the fetch process, the next mirror
+will be tried.
++
+Once the fetch from the mirror is complete, the central host is
+contacted and fetched from.  If the mirror was correct and up to date,
+then no more data will be required from the central host.  At this
+point, all of the refs under `refs/mirrors/`<remote> which are
+reachable from the real `refs/remotes/`<remote> tracking branches will
+be removed.  Extra refs which were not present on the real source will
+be left behind so they are not fetched again the next time around.
++
+This can be made the default for a remote using the
+`remote.`<remote>.`use-mirror` configuration option.
+
 -v::
 --verbose::
 	Be verbose.
diff --git a/builtin-fetch.c b/builtin-fetch.c
index a35a6f8..209f502 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -23,7 +23,7 @@ enum {
 	TAGS_SET = 2
 };
 
-static int append, force, keep, update_head_ok, verbosity;
+static int append, force, keep, update_head_ok, verbosity, use_mirror;
 static int tags = TAGS_DEFAULT;
 static const char *depth;
 static const char *upload_pack;
@@ -45,6 +45,8 @@ static struct option builtin_fetch_options[] = {
 	OPT_BOOLEAN('k', "keep", &keep, "keep downloaded pack"),
 	OPT_BOOLEAN('u', "update-head-ok", &update_head_ok,
 		    "allow updating of HEAD ref"),
+	OPT_BOOLEAN('M', "mirror", &use_mirror,
+		    "use mirror if available"),
 	OPT_STRING(0, "depth", &depth, "DEPTH",
 		   "deepen history of shallow clone"),
 	OPT_END()
diff --git a/remote.c b/remote.c
index 73d33f2..65df03d 100644
--- a/remote.c
+++ b/remote.c
@@ -111,6 +111,13 @@ static void add_pushurl(struct remote *remote, const char *pushurl)
 	remote->pushurl[remote->pushurl_nr++] = pushurl;
 }
 
+static void add_mirror_url(struct remote *remote, const char *mirror_url)
+{
+	ALLOC_GROW(remote->mirror_url, remote->mirror_url_nr + 1,
+		   remote->mirror_url_alloc);
+	remote->mirror_url[remote->mirror_url_nr++] = mirror_url;
+}
+
 static void add_pushurl_alias(struct remote *remote, const char *url)
 {
 	const char *pushurl = alias_url(url, &rewrites_push);
@@ -407,6 +414,16 @@ static int handle_config(const char *key, const char *value, void *cb)
 		if (git_config_string(&v, key, value))
 			return -1;
 		add_pushurl(remote, v);
+	} else if (!strcmp(subkey, ".mirror-url")) {
+		const char *v;
+		if (git_config_string(&v, key, value))
+			return -1;
+		add_mirror_url(remote, v);
+	} else if (!strcmp(subkey, ".use-mirror")) {
+		remote->use_mirror = git_config_bool(key, value);
+	} else if (!strcmp(subkey, ".preferred-mirror")) {
+		if (git_config_string(&remote->preferred_mirror, key, value))
+			return -1;
 	} else if (!strcmp(subkey, ".push")) {
 		const char *v;
 		if (git_config_string(&v, key, value))
diff --git a/remote.h b/remote.h
index 5db8420..c720b9a 100644
--- a/remote.h
+++ b/remote.h
@@ -19,6 +19,12 @@ struct remote {
 	int pushurl_nr;
 	int pushurl_alloc;
 
+	const char **mirror_url;
+	int mirror_url_nr;
+	int mirror_url_alloc;
+	int use_mirror;
+	const char *preferred_mirror;
+
 	const char **push_refspec;
 	struct refspec *push;
 	int push_refspec_nr;
-- 
1.6.3.3

[PATCH 2/4] fetch: try mirrors if selected

[Sam Vilain]

If configured and selected, the mirrors are tried in turn until one
succeeds, re-writing the refs to a refs/mirrors/<remote>/<hostname>/ space.
No refs from the mirrors are ever written to the real refs/heads or
refs/tags spaces, but their being available locally will speed up fetching
from the real remote if they are more up to date than the local version.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 builtin-fetch.c         |  161 ++++++++++++++++++++++++++++++++++++++++++++--
 remote.c                |   14 ++++-
 remote.h                |    1 +
 t/t5560-mirror-fetch.sh |   46 +++++++++++++
 transport.c             |   41 ++++++++++++
 transport.h             |    5 ++
 6 files changed, 260 insertions(+), 8 deletions(-)
 create mode 100644 t/t5560-mirror-fetch.sh

diff --git a/builtin-fetch.c b/builtin-fetch.c
index 209f502..b3b8766 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -45,8 +45,8 @@ static struct option builtin_fetch_options[] = {
 	OPT_BOOLEAN('k', "keep", &keep, "keep downloaded pack"),
 	OPT_BOOLEAN('u', "update-head-ok", &update_head_ok,
 		    "allow updating of HEAD ref"),
-	OPT_BOOLEAN('M', "mirror", &use_mirror,
-		    "use mirror if available"),
+	OPT_SET_INT('M', "use-mirror", &use_mirror,
+		    "use mirror if available", 1),
 	OPT_STRING(0, "depth", &depth, "DEPTH",
 		   "deepen history of shallow clone"),
 	OPT_END()
@@ -109,6 +109,109 @@ static void find_non_local_tags(struct transport *transport,
 			struct ref **head,
 			struct ref ***tail);
 
+char* get_url_hostname(const char *url)
+{
+	char *scratch = xstrdup(url);
+	char *host = strstr(url, "://");
+	char c;
+	char *end, *rh;
+	if (host) {
+		host += 3;
+		c = '/';
+	}
+	else {
+		host = scratch;
+		c = ':';
+	}
+
+	if (host[0] == '[') {
+		end = strchr(host + 1, ']');
+		if (end) {
+			*end = 0;
+			host++;
+		}
+	}
+	else {
+		end = strchr(host, c);
+		if (end && !has_dos_drive_prefix(url) ) {
+			*end = 0;
+		}
+		else {
+			host = "localhost";
+		}
+	}
+	rh = xstrdup(host);
+	free(scratch);
+	return rh;
+}
+
+const char *mirror_ref(const char* remote_name, const char* mirror_hostname,
+		       const char* refname)
+{
+	int has_refs, new_sz;
+	char *rv, *dst;
+
+	// *rs[i] = *refspec[i];  ?
+	has_refs = ( strstr(refname, "refs/") == refname );
+	/* "refs/"(0 or 5) "mirrors/"(8) remote "/"(1) hostname "/"(1) */
+	new_sz = (has_refs ? 0 : 5) + 8
+		+ strlen(remote_name) + 1
+		+ strlen(mirror_hostname) + 1
+		+ strlen(refname) + 1;
+	rv = xmalloc( new_sz );
+	strcpy(rv, "refs/mirrors/");
+	dst = rv + 13;
+	strcpy(dst, remote_name);
+	dst += strlen(remote_name);
+	*dst++ = '/';
+	strcpy(dst, mirror_hostname);
+	dst += strlen(mirror_hostname);
+	*dst++ = '/';
+	strcpy(dst, refname+(has_refs?5:0));
+	return rv;
+}
+
+struct ref *mirror_refmap(struct transport* transport,
+			  struct ref* ref_map)
+{
+	struct ref *rm, *mirror_refmap, *last, *rv, *peer_ref;
+
+	const char* remote_name = transport->remote->name;
+	const char* mirror_hostname = get_url_hostname(transport->url);
+	int c = 0;
+
+	last = NULL;
+	rv = NULL;
+	for (rm = ref_map; rm; rm = rm->next) {
+		const char *new_dst;
+
+		// skip refs we already have locally, to avoid ref churn
+		if (has_sha1_file(rm->old_sha1))
+			continue;
+
+		mirror_refmap = alloc_ref(rm->name);
+		mirror_refmap->remote_status = rm->remote_status;
+		hashcpy(mirror_refmap->old_sha1, rm->old_sha1);
+		hashcpy(mirror_refmap->new_sha1, rm->new_sha1);
+
+		if (last)
+			last->next = mirror_refmap;
+		else
+			rv = mirror_refmap;
+		c++;
+
+		new_dst = mirror_ref(remote_name, mirror_hostname, rm->name);
+
+		peer_ref = alloc_ref(new_dst);
+		mirror_refmap->peer_ref = peer_ref;
+		peer_ref->force = 1;
+		last = mirror_refmap;
+	}
+
+	return rv;
+}
+
+
 static struct ref *get_ref_map(struct transport *transport,
 			       struct refspec *refs, int ref_count, int tags,
 			       int *autotags)
@@ -165,10 +268,14 @@ static struct ref *get_ref_map(struct transport *transport,
 	if (tags == TAGS_DEFAULT && *autotags)
 		find_non_local_tags(transport, &ref_map, &tail);
 	ref_remove_duplicates(ref_map);
+	if (strcmp(transport->url, transport->remote->url[0]) != 0) {
+		return mirror_refmap(transport, ref_map);
+	}
 
 	return ref_map;
 }
 
+
 #define STORE_REF_ERROR_OTHER 1
 #define STORE_REF_ERROR_DF_CONFLICT 2
 
@@ -638,6 +745,7 @@ static int do_fetch(struct transport *transport,
 	}
 
 	ref_map = get_ref_map(transport, refs, ref_count, tags, &autotags);
+
 	if (!update_head_ok)
 		check_not_current_branch(ref_map);
 
@@ -688,13 +796,18 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	int i;
 	static const char **refs = NULL;
 	int ref_nr = 0;
-	int exit_code;
+	int exit_code = 0;
+	int urls_remaining = 1;
+	struct transport *real_transport = NULL;
+	const char *mirror = NULL;
+	struct refspec *refspec;
 
 	/* Record the command line for the reflog */
 	strbuf_addstr(&default_rla, "fetch");
 	for (i = 1; i < argc; i++)
 		strbuf_addf(&default_rla, " %s", argv[i]);
 
+	use_mirror = -1;
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
@@ -706,6 +819,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	if (!remote)
 		die("Where do you want to fetch from today?");
 
+	if (use_mirror == -1) {
+		use_mirror = remote->use_mirror ? 1 : 0;
+	}
+
 	transport = transport_get(remote, remote->url[0]);
 	if (verbosity >= 2)
 		transport->verbose = 1;
@@ -742,9 +859,39 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack);
-	exit_code = do_fetch(transport,
-			parse_fetch_refspec(ref_nr, refs), ref_nr);
-	transport_disconnect(transport);
-	transport = NULL;
+	refspec = parse_fetch_refspec(ref_nr, refs);
+	if (use_mirror) {
+		real_transport = transport;
+		urls_remaining = remote->mirror_url_nr + 1;
+	}
+	while (urls_remaining) {
+		if (use_mirror && (urls_remaining > 1) ) {
+			transport = transport_next_mirror(real_transport, mirror);
+			mirror = transport->url;
+			warning("trying mirror: %s", mirror);
+			// real_transport may not have these options - re-set them.
+			if (upload_pack)
+				set_option(TRANS_OPT_UPLOADPACK, upload_pack);
+			if (keep)
+				set_option(TRANS_OPT_KEEP, "yes");
+			if (depth)
+				set_option(TRANS_OPT_DEPTH, depth);
+
+		}
+		exit_code = do_fetch(transport, refspec, ref_nr);
+		transport_disconnect(transport);
+		transport = NULL;
+		urls_remaining--;
+		if (use_mirror) {
+			if (!exit_code && urls_remaining >= 1) {
+				warning("successful fetch from mirror");
+				urls_remaining = 1;
+			}
+			if (urls_remaining == 1) {
+				transport = real_transport;
+				warning("trying master: %s", transport->url);
+			}
+		}
+	}
 	return exit_code;
 }
diff --git a/remote.c b/remote.c
index 65df03d..5f08e10 100644
--- a/remote.c
+++ b/remote.c
@@ -139,8 +139,9 @@ static struct remote *make_remote(const char *name, int len)
 	for (i = 0; i < remotes_nr; i++) {
 		if (len ? (!strncmp(name, remotes[i]->name, len) &&
 			   !remotes[i]->name[len]) :
-		    !strcmp(name, remotes[i]->name))
+		    !strcmp(name, remotes[i]->name)) {
 			return remotes[i];
+		}
 	}
 
 	ret = xcalloc(1, sizeof(struct remote));
@@ -683,6 +684,7 @@ static struct refspec *parse_push_refspec(int nr_refspec, const char **refspec)
 	return parse_refspec_internal(nr_refspec, refspec, 0, 0);
 }
 
+
 static int valid_remote_nick(const char *name)
 {
 	if (!name[0] || is_dot_or_dotdot(name))
@@ -786,6 +788,16 @@ int remote_has_url(struct remote *remote, const char *url)
 	return 0;
 }
 
+int remote_mirror_idx(struct remote *remote, const char *mirror_url)
+{
+	int i;
+	for (i = 0; i < remote->mirror_url_nr; i++) {
+		if (!strcmp(remote->mirror_url[i], mirror_url))
+			return i;
+	}
+	return -1;
+}
+
 static int match_name_with_pattern(const char *key, const char *name,
 				   const char *value, char **result)
 {
diff --git a/remote.h b/remote.h
index c720b9a..da208ff 100644
--- a/remote.h
+++ b/remote.h
@@ -61,6 +61,7 @@ typedef int each_remote_fn(struct remote *remote, void *priv);
 int for_each_remote(each_remote_fn fn, void *priv);
 
 int remote_has_url(struct remote *remote, const char *url);
+int remote_mirror_idx(struct remote *remote, const char *mirror_url);
 
 struct refspec {
 	unsigned force : 1;
diff --git a/t/t5560-mirror-fetch.sh b/t/t5560-mirror-fetch.sh
new file mode 100644
index 0000000..940dc0e
--- /dev/null
+++ b/t/t5560-mirror-fetch.sh
@@ -0,0 +1,46 @@
+#!/bin/sh
+#
+# Copyright (c) 2009 Sam Vilain
+#
+
+test_description='mirror fetch test'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	echo >file master initial &&
+	git add file &&
+	git commit -a -m "Master initial" &&
+	git clone . master &&
+	git clone master mirror &&
+	cd master &&
+	echo >file master update &&
+	git commit -a -m "Master update" &&
+	cd .. &&
+	mkdir clone &&
+	cd clone &&
+	git init &&
+	git remote add origin ../master &&
+	git config remote.origin.mirror-url ../mirror
+'
+
+# in later iterations we'll expect these mirror tracking refs to be
+# cleaned up once they are confirmed reachable from the master, but
+# for now they leave a sufficient breadcrumb of the operation
+
+test_expect_success 'fetch using mirror - explicit' '
+	git fetch --use-mirror origin refs/heads/*:refs/remotes/origin/* &&
+	git rev-parse refs/mirrors/origin/localhost/heads/master
+'
+
+test_expect_success 'fetch using mirror - default' '
+	cd .. &&
+	mkdir clone2 &&
+	cd clone2 &&
+	git init &&
+	git remote add origin ../master &&
+	git config remote.origin.mirror-url ../mirror
+	git fetch --use-mirror &&
+	git rev-parse refs/mirrors/origin/localhost/heads/master
+'
+test_done
diff --git a/transport.c b/transport.c
index 644a30a..0dc0185 100644
--- a/transport.c
+++ b/transport.c
@@ -859,6 +859,47 @@ struct transport *transport_get(struct remote *remote, const char *url)
 	return ret;
 }
 
+struct transport *transport_next_mirror(struct transport *transport,
+					const char *last_mirror)
+{
+	struct transport *ret;
+	struct remote* remote = transport->remote;
+	int mirror_idx = -1;
+	const char* url;
+
+	if (!last_mirror) {
+		if (remote->preferred_mirror) {
+			mirror_idx = remote_mirror_idx(
+				remote,
+				remote->preferred_mirror
+				);
+			if (mirror_idx == -1) {
+				warning("preferred mirror '%s' not listed "
+					"in remote.%s.mirror-url",
+					remote->preferred_mirror,
+					remote->name);
+			}
+		}
+		else {
+			mirror_idx = 0;
+		}
+	}
+	else {
+		mirror_idx = remote_mirror_idx(remote, last_mirror) + 1;
+		// caller must check that we are not looping indefinitely
+		mirror_idx %= remote->mirror_url_nr;
+	}
+
+	url = remote->mirror_url[mirror_idx];
+	ret = transport_get(remote, url);
+
+	// copy settings - caller must re-set options
+	ret->verbose = transport->verbose;
+	ret->progress = transport->progress;
+
+	return ret;
+}
+
 int transport_set_option(struct transport *transport,
 			 const char *name, const char *value)
 {
diff --git a/transport.h b/transport.h
index c14da6f..9890157 100644
--- a/transport.h
+++ b/transport.h
@@ -41,6 +41,9 @@ struct transport {
 /* Returns a transport suitable for the url */
 struct transport *transport_get(struct remote *, const char *);
 
+/* Returns a transport for a mirror */
+struct transport *transport_next_mirror(struct transport *transport, const char *last_mirror);
+
 /* Transport options which apply to git:// and scp-style URLs */
 
 /* The program to use on the remote side to send a pack */
@@ -78,6 +81,8 @@ int transport_fetch_refs(struct transport *transport, const struct ref *refs);
 void transport_unlock_pack(struct transport *transport);
 int transport_disconnect(struct transport *transport);
 char *transport_anonymize_url(const char *url);
+struct refspec *mirror_refspec(struct transport* transport,
+			       struct refspec *refspec, int refspec_nr);
 
 /* Transport methods defined outside transport.c */
 int transport_helper_init(struct transport *transport, const char *name);
-- 
1.6.3.3

[PATCH 3/4] fetch --use-mirror: don't fetch with 'autotags' for actual fetch

[Sam Vilain]

Unsetting 'autotags' at a late stage during the fetch process has the
useful behaviour of figuring out which refs to fetch where according to the
regular autotags rules, building a refspec (struct ref* linked list), and
then we turn them off for mirror fetch and no real tags are actually
changed, just the re-written ones under refs/mirrors/.  The final fetch
will re-set autotags again, and uncannily the exact behaviour we are after
springs up: we get all the tags for the refs that are now changing, even
though we got the data from a mirror.  All from one line of code.  Win!

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 builtin-fetch.c         |    1 +
 t/t5560-mirror-fetch.sh |   12 +++++++++++-
 2 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/builtin-fetch.c b/builtin-fetch.c
index b3b8766..daa287a 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -269,6 +269,7 @@ static struct ref *get_ref_map(struct transport *transport,
 		find_non_local_tags(transport, &ref_map, &tail);
 	ref_remove_duplicates(ref_map);
 	if (strcmp(transport->url, transport->remote->url[0]) != 0) {
+		*autotags = 0;
 		return mirror_refmap(transport, ref_map);
 	}
 
diff --git a/t/t5560-mirror-fetch.sh b/t/t5560-mirror-fetch.sh
index 940dc0e..58d5f3c 100644
--- a/t/t5560-mirror-fetch.sh
+++ b/t/t5560-mirror-fetch.sh
@@ -11,11 +11,13 @@ test_expect_success setup '
 	echo >file master initial &&
 	git add file &&
 	git commit -a -m "Master initial" &&
+	git tag -m "SEEN" initial &&
 	git clone . master &&
 	git clone master mirror &&
 	cd master &&
 	echo >file master update &&
 	git commit -a -m "Master update" &&
+	git tag -m "SEEN" update &&
 	cd .. &&
 	mkdir clone &&
 	cd clone &&
@@ -35,12 +37,20 @@ test_expect_success 'fetch using mirror - explicit' '
 
 test_expect_success 'fetch using mirror - default' '
 	cd .. &&
+	cd mirror &&
+	git tag -m "badtag" badtag &&
+	cd .. &&
 	mkdir clone2 &&
 	cd clone2 &&
 	git init &&
 	git remote add origin ../master &&
 	git config remote.origin.mirror-url ../mirror
 	git fetch --use-mirror &&
-	git rev-parse refs/mirrors/origin/localhost/heads/master
+	git rev-parse refs/mirrors/origin/localhost/heads/master &&
+	git rev-parse refs/mirrors/origin/localhost/tags/initial &&
+	! git rev-parse refs/tags/badtag &&
+	git rev-parse refs/tags/initial &&
+	git rev-parse refs/tags/update
 '
+
 test_done
-- 
1.6.3.3

[PATCH 4/4] fetch: cleanup refs with --use-mirror

[Sam Vilain]

Remove identical refs after a successful fetch.  The ref under
'refs/mirrors/HOST/XXX' is compared with 'refs/XXX', and if matched,
then the 'refs/mirrors/' version is removed.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
  This is a simple mechanism for removing stale mirror refs; a more
  sophisticated approach would use the revision walker.

 builtin-fetch.c |   62 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 59 insertions(+), 3 deletions(-)

diff --git a/builtin-fetch.c b/builtin-fetch.c
index daa287a..0c52f23 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -211,6 +211,57 @@ struct ref *mirror_refmap(struct transport* transport,
 	return rv;
 }
 
+int clean_up_mirror_ref(const char *refname,
+			const unsigned char *sha1,
+			int flags,
+			void *leading)
+{
+	char *orig_refname;
+	char *target_refname;
+	char *x;
+	unsigned char found_sha1[20];
+
+	orig_refname = xmalloc(strlen(refname)+strlen(leading)+1);
+	x = strchr(refname, '/');
+	if (!x)
+		return 0;
+	target_refname = xmalloc(strlen(x)+strlen("refs/remotes/")+1); 
+
+	strcpy(orig_refname, leading);
+	x = orig_refname + strlen(leading);
+	strcpy(x, refname);
+
+	warning("cleaning up mirror ref: %s (%s)",
+		orig_refname, sha1_to_hex(sha1));
+
+	strcpy(target_refname, "refs/remotes/");
+	strcpy(target_refname+5, strchr(refname, '/')+1);
+
+	warning("target ref is %s", target_refname);
+
+	if (resolve_ref(target_refname, found_sha1, 1, NULL)) {
+		if (!hashcmp(found_sha1, sha1)) {
+			warning("deleting ref %s", orig_refname);
+			delete_ref(orig_refname, sha1, REF_NODEREF);
+		}
+	}
+}
+
+void clean_up_mirror_refs(struct remote* remote)
+{
+	int rem_l = strlen(remote->name);
+	char *dst_name = xmalloc(rem_l+14);
+	char *x;
+	strcpy(dst_name, "refs/mirrors/");
+	x = dst_name + 13;
+	strcpy(x, remote->name);
+	x += rem_l;
+	*x++ = '/';
+
+	warning("cleaning up mirror refs for remote %s", remote->name);
+	for_each_ref_in(dst_name, clean_up_mirror_ref,
+			(void *)dst_name);
+}
 
 static struct ref *get_ref_map(struct transport *transport,
 			       struct refspec *refs, int ref_count, int tags,
@@ -884,9 +935,14 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		transport = NULL;
 		urls_remaining--;
 		if (use_mirror) {
-			if (!exit_code && urls_remaining >= 1) {
-				warning("successful fetch from mirror");
-				urls_remaining = 1;
+			if (!exit_code) {
+				if (urls_remaining >= 1) {
+					warning("successful fetch from mirror");
+					urls_remaining = 1;
+				}
+				else {
+					clean_up_mirror_refs(remote);
+				}
 			}
 			if (urls_remaining == 1) {
 				transport = real_transport;
-- 
1.6.3.3

Re: [PATCH 2/4] fetch: try mirrors if selected

[Shawn O. Pearce]

Sam Vilain <sam@vilain.net> wrote:
> diff --git a/builtin-fetch.c b/builtin-fetch.c
> index 209f502..b3b8766 100644
> @@ -109,6 +109,109 @@ static void find_non_local_tags(struct transport *transport,
>  			struct ref **head,
>  			struct ref ***tail);
>  
> +char* get_url_hostname(const char *url)

Minor nit, but we mark any function not used outside of the module
as static.  Especially in a builtin-*.c since they all link into
the same namespace.  If this is meant to be reused, it belongs in
connect.c most likely, that's where we already have code like this
to get the SSH hostname out of a URL for SSH connections.

I don't have time right now to read the rest of this series, but
the general approach of fetching to a temporary mirror space before
checking if you really are current is a good one.

I'm not sure that storing the list of mirrors inside of the remote
makes much sense, I would think the user would want to store only
a handful of "fast" URLs.

And even then I wonder why this can't just be the url[1]..url[n-1]
entries in the configuration file.  push pushes to all of the
URLs at once, "seeding the mirrors".  Why can't fetch use the same
configuration?

-- 
Shawn.

Re: Client-side mirroring patches (v0)

[Shawn O. Pearce]

Sam Vilain <sam@vilain.net> wrote:
> Hey folks, this is the first stage of git mirroring 
...
> Also there is the matter of falling over to the next mirror should one
> not be reachable, but then we're getting into C weaknesses really.
> Should I plan to do exception recovery using 'longjmp' ? 

Please don't use longjmp.

You'll have to change the code to not die() upon connection failure,
but instead return an error code to the higher level which can
locate another mirror and retry.

> Also the
> process should be interruptible and provide a user menu.  Again this
> seems like it would be very tedious and clumsy in C.  How do people
> manage?

With great pain.  :-)

To do a user menu you can do a simple interface like `git add -i`
does, which just dumps the choices to stdout and a prompt for the
user to enter their selection.  If you want something more complex
you need to link to curses or ncurses, which IIRC opens some issues
with portablity, but lets you do a bit nicer interface on the tty.

-- 
Shawn.

Re: Client-side mirroring patches (v0)

[Nanako Shiraishi]

Junio, could you tell us what happened to this thread?