[PATCH 2/2] Add support for host aliases in config files

[PATCH 2/2] Add support for host aliases in config files

[Daniel Barkalow]

This allows users with different preferences for access methods to the
same remote repositories to rewrite each other's URLs by pattern
matching across a large set of similiarly set up repositories to each
get the desired access.

For example, if you don't have a kernel.org account, you might want
settings like:

[host "kernel.org"]
      base = git://git.kernel.org/pub/
      rewritebase = master.kernel.org:/pub

Then, if you give git a URL like:

  master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git

it will act like you gave it:

  git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git

and you can cut-and-paste pull requests in email without fixing them
by hand, for example.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---
I believe this takes care of all the comments I've gotten on this patch.

I looked at how the documentation appears in (for example) git-fetch.1 
with the change to urls.txt, and it looks informative to me and not 
unbalanced. Of course, in order to be perfectly accurate, git clone needs 
to be converted to using the same code as everything else, because it 
won't yet actually support these alises, unfortunately for my dreams of 
being able to set up a default host and then have "git clone :module" work.

 Documentation/config.txt |   23 ++++++++++
 Documentation/urls.txt   |   25 +++++++++++
 remote.c                 |  102 ++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 147 insertions(+), 3 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index f2f6a77..7a9cba4 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -646,6 +646,29 @@ help.format::
 	Values 'man', 'info', 'web' and 'html' are supported. 'man' is
 	the default. 'web' and 'html' are the same.
 
+host.<name>.*::
+	These options provide a way to rewrite URLs when there is a
+	pattern of URLs with a common prefix which should be replaced
+	with a different prefix. For every 'rewritebase' prefix given,
+	any URL git receives that starts with that prefix will be
+	rewritten to have the 'base' prefix instead. In cases where
+	some site serves a large number of repositories, and serves
+	them with multiple access methods, and some users need to use
+	different access methods, this feature allows people to
+	specify any of the equivalent URLs and have git automatically
+	rewrite the URL to the best alternative for the particular
+	user, even for a never-before-seen repository on the site. The
+	effect of multiple matching rewritebase prefixes is undefined.
+
+host.<name>.base::
+	The base URL which should be used for this particular
+	host.
+
+host.<name>.rewritebase::
+	Additional base URLs which refer to this host. If a URL
+	matches this, any access to it will use the URL formed with
+	the corresponding base URL instead of the given URL.
+
 http.proxy::
 	Override the HTTP proxy, normally configured using the 'http_proxy'
 	environment variable (see linkgit:curl[1]).  This can be overridden
diff --git a/Documentation/urls.txt b/Documentation/urls.txt
index 81ac17f..11c05db 100644
--- a/Documentation/urls.txt
+++ b/Documentation/urls.txt
@@ -44,3 +44,28 @@ endif::git-clone[]
 ifdef::git-clone[]
 They are equivalent, except the former implies --local option.
 endif::git-clone[]
+
+
+If there are a large number of similarly-named remote repositories and
+you want to use a different format for them (such that the URLs you
+use will be rewritten into URLs that work), you can create a
+configuration section of the form:
+
+------------
+	[host "<host>"]
+		base = <actual url base>
+		rewritebase = <other url base>
+------------
+
+If you have a section:
+
+------------
+	[host "xz"]
+		base = git://git.host.xz/
+		rewritebase = host.xz:/path/to/
+		rewritebase = work:
+------------
+
+a URL like "work:repo.git" or like "host.xz:/path/to/repo.git" will be
+rewritten in any context that takes a URL to be
+"git://git.host.xz/repo.git".
diff --git a/remote.c b/remote.c
index 457d8a4..fc83e2d 100644
--- a/remote.c
+++ b/remote.c
@@ -2,6 +2,14 @@
 #include "remote.h"
 #include "refs.h"
 
+struct host {
+	const char *name;
+	const char *base;
+	const char **alias;
+	int alias_nr;
+	int alias_alloc;
+};
+
 static struct remote **remotes;
 static int remotes_alloc;
 static int remotes_nr;
@@ -13,9 +21,33 @@ static int branches_nr;
 static struct branch *current_branch;
 static const char *default_remote_name;
 
+static struct host **hosts;
+static int hosts_alloc;
+static int hosts_nr;
+
 #define BUF_SIZE (2048)
 static char buffer[BUF_SIZE];
 
+static const char *alias_url(const char *url)
+{
+	int i, j;
+	for (i = 0; i < hosts_nr; i++) {
+		if (!hosts[i])
+			continue;
+		for (j = 0; j < hosts[i]->alias_nr; j++) {
+			if (!prefixcmp(url, hosts[i]->alias[j])) {
+				char *ret = malloc(strlen(hosts[i]->base) -
+						   strlen(hosts[i]->alias[j]) +
+						   strlen(url) + 1);
+				strcpy(ret, hosts[i]->base);
+				strcat(ret, url + strlen(hosts[i]->alias[j]));
+				return ret;
+			}
+		}
+	}
+	return url;
+}
+
 static void add_push_refspec(struct remote *remote, const char *ref)
 {
 	ALLOC_GROW(remote->push_refspec,
@@ -38,6 +70,11 @@ static void add_url(struct remote *remote, const char *url)
 	remote->url[remote->url_nr++] = url;
 }
 
+static void add_url_alias(struct remote *remote, const char *url)
+{
+	add_url(remote, alias_url(url));
+}
+
 static struct remote *make_remote(const char *name, int len)
 {
 	struct remote *ret;
@@ -95,6 +132,35 @@ static struct branch *make_branch(const char *name, int len)
 	return ret;
 }
 
+static struct host *make_host(const char *name, int len)
+{
+	struct host *ret;
+	int i;
+
+	for (i = 0; i < hosts_nr; i++) {
+		if (len ? (!strncmp(name, hosts[i]->name, len) &&
+			   !hosts[i]->name[len]) :
+		    !strcmp(name, hosts[i]->name))
+			return hosts[i];
+	}
+
+	ALLOC_GROW(hosts, hosts_nr + 1, hosts_alloc);
+	ret = xcalloc(1, sizeof(struct host));
+	hosts[hosts_nr++] = ret;
+	if (len)
+		ret->name = xstrndup(name, len);
+	else
+		ret->name = xstrdup(name);
+
+	return ret;
+}
+
+static void add_alias(struct host *host, const char *name)
+{
+	ALLOC_GROW(host->alias, host->alias_nr + 1, host->alias_alloc);
+	host->alias[host->alias_nr++] = name;
+}
+
 static void read_remotes_file(struct remote *remote)
 {
 	FILE *f = fopen(git_path("remotes/%s", remote->name), "r");
@@ -128,7 +194,7 @@ static void read_remotes_file(struct remote *remote)
 
 		switch (value_list) {
 		case 0:
-			add_url(remote, xstrdup(s));
+			add_url_alias(remote, xstrdup(s));
 			break;
 		case 1:
 			add_push_refspec(remote, xstrdup(s));
@@ -180,7 +246,7 @@ static void read_branches_file(struct remote *remote)
 	} else {
 		branch = "refs/heads/master";
 	}
-	add_url(remote, p);
+	add_url_alias(remote, p);
 	add_fetch_refspec(remote, branch);
 	remote->fetch_tags = 1; /* always auto-follow */
 }
@@ -210,6 +276,23 @@ static int handle_config(const char *key, const char *value)
 		}
 		return 0;
 	}
+	if (!prefixcmp(key, "host.")) {
+		struct host *host;
+		name = key + 5;
+		subkey = strrchr(name, '.');
+		if (!subkey)
+			return 0;
+		host = make_host(name, subkey - name);
+		if (!strcmp(subkey, ".base")) {
+			if (!value)
+				return config_error_nonbool(key);
+			host->base = xstrdup(value);
+		} else if (!strcmp(subkey, ".rewritebase")) {
+			if (!value)
+				return config_error_nonbool(key);
+			add_alias(host, xstrdup(value));
+		}
+	}
 	if (prefixcmp(key,  "remote."))
 		return 0;
 	name = key + 7;
@@ -261,6 +344,18 @@ static int handle_config(const char *key, const char *value)
 	return 0;
 }
 
+static void alias_all_urls(void)
+{
+	int i, j;
+	for (i = 0; i < remotes_nr; i++) {
+		if (!remotes[i])
+			continue;
+		for (j = 0; j < remotes[i]->url_nr; j++) {
+			remotes[i]->url[j] = alias_url(remotes[i]->url[j]);
+		}
+	}
+}
+
 static void read_config(void)
 {
 	unsigned char sha1[20];
@@ -277,6 +372,7 @@ static void read_config(void)
 			make_branch(head_ref + strlen("refs/heads/"), 0);
 	}
 	git_config(handle_config);
+	alias_all_urls();
 }
 
 struct refspec *parse_ref_spec(int nr_refspec, const char **refspec)
@@ -342,7 +438,7 @@ struct remote *remote_get(const char *name)
 			read_branches_file(ret);
 	}
 	if (!ret->url)
-		add_url(ret, name);
+		add_url_alias(ret, name);
 	if (!ret->url)
 		return NULL;
 	ret->fetch = parse_ref_spec(ret->fetch_refspec_nr, ret->fetch_refspec);
-- 
1.5.4.1.191.gfbf10

Re: [PATCH 2/2] Add support for host aliases in config files

[Jon Loeliger]

Daniel Barkalow wrote:
> This allows users with different preferences for access methods to the
> same remote repositories to rewrite each other's URLs by pattern
> matching across a large set of similiarly set up repositories to each
> get the desired access.
> 
> For example, if you don't have a kernel.org account, you might want
> settings like:
> 
> [host "kernel.org"]
>       base = git://git.kernel.org/pub/
>       rewritebase = master.kernel.org:/pub
> 
> Then, if you give git a URL like:
> 
>   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> 
> it will act like you gave it:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> 
> and you can cut-and-paste pull requests in email without fixing them
> by hand, for example.
> 
> Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> ---

So, I may be dense, but I'm having a hard time distinguishing
the names "base" and "rewritebase" just from a User Interface
perspective.  It's not clear to me which of those is the A and
which is the B part in s/A/B/.  "Rewritebase" might be the "from"
basis, or it might be the "to" basis.  Can we come up with more
descriptive property names here?

Is the rewrite always just prefix substitution/replacement?
What if there was some generic path rewrite needed? (KISS? :-))

Also, is "host" really the right config key?  Maybe "rewritehost"?
Or is this thought to be the basis of other (perhaps not rewrite
related) properties of the same host as well?

Maybe it's just me.

Thanks,
jdl

Re: [PATCH 2/2] Add support for host aliases in config files

[Johannes Schindelin]

Hi,

On Tue, 19 Feb 2008, Jon Loeliger wrote:

> Daniel Barkalow wrote:
>
> > For example, if you don't have a kernel.org account, you might want 
> > settings like:
> > 
> > [host "kernel.org"]
> >       base = git://git.kernel.org/pub/
> >       rewritebase = master.kernel.org:/pub
> > 
> > Then, if you give git a URL like:
> > 
> >   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > it will act like you gave it:
> > 
> >   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > and you can cut-and-paste pull requests in email without fixing them 
> > by hand, for example.
> > 
> > Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> > ---
> 
> So, I may be dense, but I'm having a hard time distinguishing the names 
> "base" and "rewritebase" just from a User Interface perspective.  It's 
> not clear to me which of those is the A and which is the B part in 
> s/A/B/.  "Rewritebase" might be the "from" basis, or it might be the 
> "to" basis.  Can we come up with more descriptive property names here?
> 
> Is the rewrite always just prefix substitution/replacement?
> What if there was some generic path rewrite needed? (KISS? :-))

I just tried this:

$ git config rewrite."master.kernel.org:/pub".url \
	git://git.kernel.org/pub/

and it worked.  IOW, the config will look like this:

[rewrite "master.kernel.org:/pub"]
        url = git://git.kernel.org/pub/

So maybe this is easier to grasp?

Ciao,
Dscho

Re: [PATCH 2/2] Add support for host aliases in config files

[Daniel Barkalow]

On Tue, 19 Feb 2008, Jon Loeliger wrote:

> Daniel Barkalow wrote:
> > This allows users with different preferences for access methods to the
> > same remote repositories to rewrite each other's URLs by pattern
> > matching across a large set of similiarly set up repositories to each
> > get the desired access.
> > 
> > For example, if you don't have a kernel.org account, you might want
> > settings like:
> > 
> > [host "kernel.org"]
> >       base = git://git.kernel.org/pub/
> >       rewritebase = master.kernel.org:/pub
> > 
> > Then, if you give git a URL like:
> > 
> >   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > it will act like you gave it:
> > 
> >   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > and you can cut-and-paste pull requests in email without fixing them
> > by hand, for example.
> > 
> > Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> > ---
> 
> So, I may be dense, but I'm having a hard time distinguishing
> the names "base" and "rewritebase" just from a User Interface
> perspective.  It's not clear to me which of those is the A and
> which is the B part in s/A/B/.  "Rewritebase" might be the "from"
> basis, or it might be the "to" basis.  Can we come up with more
> descriptive property names here?

rewritebase is the base you rewrite, which follows apache usage (which is 
where somebody got the term to suggest in an earlier round). It seems like 
it should be clear; am I obfuscating it by showing the options in the more 
backwards order in the examples? If you've got an idea that people find 
more clear, I'd be happy to change it (again).

> Is the rewrite always just prefix substitution/replacement?
> What if there was some generic path rewrite needed? (KISS? :-))

It's just prefix substitution. I suppose a future version could do 
something more clever (but it would probably also use variables named 
"pattern" instead of "base", so there shouldn't be any issues).

> Also, is "host" really the right config key?  Maybe "rewritehost"?
> Or is this thought to be the basis of other (perhaps not rewrite
> related) properties of the same host as well?

I think it's likely that other properties of the host could show up. I 
suspect that uploadpack and receivepack would be useful host-wide if you 
were using a lot of ad-hoc repository names on a big site. Rewriting the 
URLs is just the itch I had with this scope.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 2/2] Add support for host aliases in config files

[Daniel Barkalow]

On Tue, 19 Feb 2008, Johannes Schindelin wrote:

> Hi,
> 
> On Tue, 19 Feb 2008, Jon Loeliger wrote:
> 
> > Daniel Barkalow wrote:
> >
> > > For example, if you don't have a kernel.org account, you might want 
> > > settings like:
> > > 
> > > [host "kernel.org"]
> > >       base = git://git.kernel.org/pub/
> > >       rewritebase = master.kernel.org:/pub
> > > 
> > > Then, if you give git a URL like:
> > > 
> > >   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > > 
> > > it will act like you gave it:
> > > 
> > >   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > > 
> > > and you can cut-and-paste pull requests in email without fixing them 
> > > by hand, for example.
> > > 
> > > Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> > > ---
> > 
> > So, I may be dense, but I'm having a hard time distinguishing the names 
> > "base" and "rewritebase" just from a User Interface perspective.  It's 
> > not clear to me which of those is the A and which is the B part in 
> > s/A/B/.  "Rewritebase" might be the "from" basis, or it might be the 
> > "to" basis.  Can we come up with more descriptive property names here?
> > 
> > Is the rewrite always just prefix substitution/replacement?
> > What if there was some generic path rewrite needed? (KISS? :-))
> 
> I just tried this:
> 
> $ git config rewrite."master.kernel.org:/pub".url \
> 	git://git.kernel.org/pub/
> 
> and it worked.  IOW, the config will look like this:
> 
> [rewrite "master.kernel.org:/pub"]
>         url = git://git.kernel.org/pub/
> 
> So maybe this is easier to grasp?

If you've also got people using http:// URLs, this would require giving 
the preferred URL twice. And it doesn't help with possible future 
host-based configuration, or a non-prefix pattern, if that came up some 
day.

I suppose perhaps:

[host "git://git.kernel.org/pub/"]
	aka = "master.kernel.org:/pub/"
	aka = "http://www.kernel.org/pub/"

I'm a little worried about putting URLs in keys, but I don't know if 
that's a reasonable worry.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 2/2] Add support for host aliases in config files

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> On Tue, 19 Feb 2008, Jon Loeliger wrote:
>
>> Also, is "host" really the right config key?  Maybe "rewritehost"?
>> Or is this thought to be the basis of other (perhaps not rewrite
>> related) properties of the same host as well?
>
> I think it's likely that other properties of the host could show up. I 
> suspect that uploadpack and receivepack would be useful host-wide if you 
> were using a lot of ad-hoc repository names on a big site. Rewriting the 
> URLs is just the itch I had with this scope.

I'd agree that per-host configuration may be useful, and your
"paths to programs" may be a good example.  But I think it is
plausible that a host needs multiple <rewritefrom,rewriteto>
tuples.  Either your '[host "$this_url"] aka = $that_url' or
Dscho's earlier '[rewrite "$this_url_to"] url = $that_url' shows
that this is not about host but "prefix part of URL".

Re: [PATCH 2/2] Add support for host aliases in config files

[Daniel Barkalow]

On Tue, 19 Feb 2008, Junio C Hamano wrote:

> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > On Tue, 19 Feb 2008, Jon Loeliger wrote:
> >
> >> Also, is "host" really the right config key?  Maybe "rewritehost"?
> >> Or is this thought to be the basis of other (perhaps not rewrite
> >> related) properties of the same host as well?
> >
> > I think it's likely that other properties of the host could show up. I 
> > suspect that uploadpack and receivepack would be useful host-wide if you 
> > were using a lot of ad-hoc repository names on a big site. Rewriting the 
> > URLs is just the itch I had with this scope.
> 
> I'd agree that per-host configuration may be useful, and your
> "paths to programs" may be a good example.  But I think it is
> plausible that a host needs multiple <rewritefrom,rewriteto>
> tuples.

That's probably true, even taking into account the option of having 
multiple froms for the same to.

> Either your '[host "$this_url"] aka = $that_url' or
> Dscho's earlier '[rewrite "$this_url_to"] url = $that_url' shows
> that this is not about host but "prefix part of URL".

Well, it shows that it's *possible* to do this feature that way, although 
I'm not entirely convinced that it's best to do it like that. Are there 
character set concerns with having urls in config keys?

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 2/2] Add support for host aliases in config files

[Johannes Schindelin]

Hi,

On Tue, 19 Feb 2008, Daniel Barkalow wrote:

> On Tue, 19 Feb 2008, Junio C Hamano wrote:
> 
> > Either your '[host "$this_url"] aka = $that_url' or
> > Dscho's earlier '[rewrite "$this_url_to"] url = $that_url' shows
> > that this is not about host but "prefix part of URL".
> 
> Well, it shows that it's *possible* to do this feature that way, 
> although I'm not entirely convinced that it's best to do it like that. 

I think in the interest of simplicity, you do not want to have an extra 
(arbitrary) key.

> Are there character set concerns with having urls in config keys?

I think not.  AFAICT the "extended base var" is allowed anything except 
for newlines.  Everything else you can get at least escaped with '\'.  If 
in doubt, consult get_extended_base_var() in config.c.

Hth,
Dscho

Re: [PATCH 2/2] Add support for host aliases in config files

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> ...  If 
> in doubt, consult get_extended_base_var() in config.c.

Perhaps it is time for Documentation/technical/api-config.txt
;-)?

Re: [PATCH 2/2] Add support for host aliases in config files

[Johannes Schindelin]

Hi,

On Tue, 19 Feb 2008, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > ...  If in doubt, consult get_extended_base_var() in config.c.
> 
> Perhaps it is time for Documentation/technical/api-config.txt ;-)?

*winces* I try to find the time...

Sorry,
Dscho

Re: [PATCH 2/2] Add support for host aliases in config files

[Jakub Narebski]

Johannes Schindelin wrotr:
 
> I just tried this:
> 
> $ git config rewrite."master.kernel.org:/pub".url \
> 	git://git.kernel.org/pub/
> 
> and it worked.  IOW, the config will look like this:
> 
> [rewrite "master.kernel.org:/pub"]
>         url = git://git.kernel.org/pub/
> 
> So maybe this is easier to grasp?

Perhaps that it is easier to grasp, but imho it is backwards,
as it is inherently 1 -> 1 in <original> -> <rewritten>,
while I think it is more common to have n -> 1, i.e. multiple
URLs (aliases), multiple alternative "spellings" with one
tried way to access repository.

-- 
Jakub Narebski
Poland