[PATCH] git-remote exclude

[PATCH] git-remote exclude

[Quy Tonthat]

"git-remote exclude" can be used to prevent one or more unwanted remote branches
from being tracked. After, for example,
    $ git-remote origin exclude man html
"git-fetch origin" will no longer fetch origin/man and origin/html.

"git-remote exclude" does not check whether the "unwanted" branch
really exists on the remote repo.

Signed-off-by: Quy Tonthat <qtonthat@gmail.com>
---
 git-remote.perl |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 51 insertions(+), 0 deletions(-)

diff --git a/git-remote.perl b/git-remote.perl
index fc055b6..8ee1bea 100755
--- a/git-remote.perl
+++ b/git-remote.perl
@@ -243,6 +243,47 @@ sub add_remote {
 		      "+refs/heads/*:refs/remotes/$name/*");
 }
 
+sub exclude_remote_branch {
+	my ($name, @branches) = @_;
+	if (!exists $remote->{$name}) {
+		print STDERR "remote $name does not exists.\n";
+		exit(1);
+	}
+
+	my $wildcard = "";
+	my %fetch_entries = ();
+	my @fetches = $git->command(qw(repo-config --get-all), "remote.$name.fetch");
+	foreach $fetch (@fetches) {
+		if ($fetch =~ m%^\+?refs/heads/\*:refs/remotes/$name/\*$%) {
+			$wildcard = $fetch;
+		}
+		else {
+			foreach $br (@branches) {
+				if ($fetch =~ m#^refs/heads/$br:.*$#) {
+					$fetch_entries{$br} = $fetch;
+				}
+			}
+		}
+	}
+
+	if (!$wildcard) {
+		print STDERR "No wildcard entry for $name. Config not changed\n";
+		exit(1);
+	}
+	my $clean;
+       	($clean = $wildcard) =~ s/([+*])/\\$1/g;
+	$git->command(qw(repo-config --unset), "remote.$name.fetch", $clean);
+	foreach $key (keys %fetch_entries) {
+		$git->command(qw(repo-config --unset), "remote.$name.fetch",
+				$fetch_entries{$key});
+	}
+	foreach $br (@branches) {
+		$git->command(qw(repo-config --add), "remote.$name.fetch",
+			"refs/heads/$br:");
+	}
+	$git->command(qw(repo-config --add), "remote.$name.fetch", $wildcard);
+}
+
 if (!@ARGV) {
 	for (sort keys %$remote) {
 		print "$_\n";
@@ -274,9 +315,19 @@ elsif ($ARGV[0] eq 'add') {
 	}
 	add_remote($ARGV[1], $ARGV[2]);
 }
+elsif ($ARGV[0] eq 'exclude') {
+	if (@ARGV < 3) {
+		print STDERR "Usage: git remote exclude <name> <branch>...\n";
+		exit(1);
+	}
+	my @branches = @ARGV;
+	shift(@branches); shift(@branches);
+	exclude_remote_branch($ARGV[1], @branches);
+}
 else {
 	print STDERR "Usage: git remote\n";
 	print STDERR "       git remote add <name> <url>\n";
+	print STDERR "       git remote exclude <name> <branch>...\n";
 	print STDERR "       git remote show <name>\n";
 	exit(1);
 }

Re: [PATCH] git-remote exclude

[Johannes Schindelin]

Hi,

On Mon, 15 Jan 2007, Quy Tonthat wrote:

> "git-remote exclude" can be used to prevent one or more unwanted remote 
> branches from being tracked. After, for example,
>     $ git-remote origin exclude man html
> "git-fetch origin" will no longer fetch origin/man and origin/html.

That is not what your patch does.

It rewrites the "remote.$name.fetch" entries so that those branches are 
not _updated_, but they are _fetched_ nevertheless.

But then, I don't really see _why_ you would want such a solution. After 
all, you are more likely to be interested in _specific_ branches, rather 
than all branches _except_ a few.

IMHO "git remote expand <name>" to expand the wildcards and "git remote 
copy <dest> <src>" would be more useful.

Ciao,
Dscho

Re: [PATCH] git-remote exclude

[Andy Parkins]

On Monday 2007 January 15 10:23, Johannes Schindelin wrote:

> But then, I don't really see _why_ you would want such a solution. After
> all, you are more likely to be interested in _specific_ branches, rather
> than all branches _except_ a few.

That's not true.  I have a patch (that doesn't work, so it's not submitted), 
that would allow me to have:

[remote "origin"]
    url = git://git2.kernel.org/pub/scm/git/git.git
    fetch = refs/heads/*:refs/remotes/up/*
    fetch = !refs/heads/html
    fetch = !refs/heads/todo

i.e. I don't want those two branches, but I do want everything else.  I'd also 
like to be able to do "!/refs/heads/temp/*" so I could block a whole 
subdirectory of branches.  I have a feeling that this would come in handy for 
people like the person who was recently talking about having 880 branches in 
his repository, with only a few active.


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIEE
andyparkins@gmail.com

Re: [PATCH] git-remote exclude

[Jakub Narebski]

Andy Parkins wrote:

> On Monday 2007 January 15 10:23, Johannes Schindelin wrote:
> 
>> But then, I don't really see _why_ you would want such a solution. After
>> all, you are more likely to be interested in _specific_ branches, rather
>> than all branches _except_ a few.
> 
> That's not true.  I have a patch (that doesn't work, so it's not submitted), 
> that would allow me to have:
> 
> [remote "origin"]
>     url = git://git2.kernel.org/pub/scm/git/git.git
>     fetch = refs/heads/*:refs/remotes/up/*
>     fetch = !refs/heads/html
>     fetch = !refs/heads/todo
> 
> i.e. I don't want those two branches, but I do want everything else.  I'd also 
> like to be able to do "!/refs/heads/temp/*" so I could block a whole 
> subdirectory of branches.  I have a feeling that this would come in handy for 
> people like the person who was recently talking about having 880 branches in 
> his repository, with only a few active.

Very good idea. I even thought that this feature is present in git
already...
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [PATCH] git-remote exclude

[Junio C Hamano]

Andy Parkins <andyparkins@gmail.com> writes:

> On Monday 2007 January 15 10:23, Johannes Schindelin wrote:
>
>> But then, I don't really see _why_ you would want such a solution. After
>> all, you are more likely to be interested in _specific_ branches, rather
>> than all branches _except_ a few.
>
> That's not true.  I have a patch (that doesn't work, so it's not submitted), 
> that would allow me to have:
>
> [remote "origin"]
>     url = git://git2.kernel.org/pub/scm/git/git.git
>     fetch = refs/heads/*:refs/remotes/up/*
>     fetch = !refs/heads/html
>     fetch = !refs/heads/todo
>
> i.e. I don't want those two branches, but I do want everything
> else.  I'd also like to be able to do "!/refs/heads/temp/*" so
> I could block a whole subdirectory of branches.

I do not think you are interested in "not these two but
everything else".  You are interested in maint, master, next, pu
and man.  You can get away by saying "not these two but
everything else" only because you are implicitly trusting me not
to publish insane number of random throw-away branches left and
right.

In an ideal world, I think you would want to be able to do
something like this:

 - Let's look at what they have right now.
 - I want, among the above, this, that and that.
 - I would never want any of the others I just checked.

 - If there are new ones found in the future, please let me know
   and I'll decide then.  or...
 - If there are new ones found in the future, I would not want to
   be bothered with them.  or...
 - If there are new ones found in the future, I want them too.

The choice between the last three will heavily depend on the
nature of the project and also your work habit to a certain
degree.

A remote could frequently throw in random small-topic branches
that are short lived, and are totally uninteresting for somebody
who wants to track the big picture branches.  You would say the
first if you primarily work in a particular area of the project,
because you will always be interested in the primary integration
and you may or may not be interested in a new topic.  You would
say the second if you are only interested in the big picture
branches after marking the primary integration branches as
interesting.  You would say the third only if you are a packrat,
or have unlimited brain bandwidth to keep track of what's
happening in every little corner of the project.

If a single repository has 1000 branches, that is a sign that
there is something wrong in the project's organization.  No
single person should need to look at 1000 branches -- that goes
directly against the distributed nature of git.  Humans do not
scale well.

Now, 'git-fetch' currently does not scale either, and I think it
is a problem, but I think that is fixable by redoing the part
below the while loop that calls append_fetch_head to make the
processing more efficient.

But you cannot optimize humans the same way.

The move to BK formalized a change to the development process of
the kernel project --- a central single integrator did not scale
well, and there was a need for a group of people around the
center to help the peer review and the integration process.  The
patch-flow in the development process places more integration
work and responsibility on the lieutenants, and the tool helps
people who are more central than the lieutenants trust, verify
and integrate their lieutenants' work.

Helping this kind of project organization is what git inherited
from BK as well.  The key to scaling is to limit the product of
the level of details times the size of the area one person needs
to look at.  A project may have 1000 simultaneous development
going at any one time, but a lieutenant should not need to look
at all 1000 branches but only a much smaller portion of them in
his own area of expertise.  A more central person should not
need to look at all 1000 branches either, but what he will look
at would be branches that the lieutenants would publish in their
repositories, as the result of integration of these 1000
branches at the lowest level.

Re: [PATCH] git-remote exclude

[Andy Parkins]

On Monday 2007 January 15 21:56, Junio C Hamano wrote:

> I do not think you are interested in "not these two but
> everything else".  You are interested in maint, master, next, pu
> and man.  You can get away by saying "not these two but
> everything else" only because you are implicitly trusting me not
> to publish insane number of random throw-away branches left and
> right.

Nope.  I think I'm right.  You're right that I'm trusting you not to publish 
an insane number of branches, but that holds for the glob-mode only, 
regardless of whether the NOT fetches are in place.  The sort of thing I 
imagine happening is that as maintainer you suddenly say - I think 1.5.0 is 
so good that we'll maintain it for longer than usual.  In which case we would 
suddenly get a branch for maint-1.5.0.  I would want that branch, but I still 
don't want man, html and todo.

>  - If there are new ones found in the future, please let me know
>    and I'll decide then.  or...
>  - If there are new ones found in the future, I would not want to
>    be bothered with them.  or...
>  - If there are new ones found in the future, I want them too.

 - If there is one that was there but now isn't, ask me if I want to remove
   it

> The choice between the last three will heavily depend on the
> nature of the project and also your work habit to a certain
> degree.

True; so all those should be possible to specify in the config.

> A remote could frequently throw in random small-topic branches
> that are short lived, and are totally uninteresting for somebody
> who wants to track the big picture branches.  You would say the

That's true.  However, if there were discipline in the naming, this wouldn't 
cause a major problem:

 fetch = refs/heads/export/*:refs/remotes/origin/*

Now, only the branches under export/ will be sent across automatically.  You 
can make whatever topic branches you want without polluting the automatic 
namespace.

> first if you primarily work in a particular area of the project,
> because you will always be interested in the primary integration
> and you may or may not be interested in a new topic.  You would

Again, in a project of that size, the branches could be conventionally stored 
in particular namespaces.

  fetch = refs/heads/topicA/*:refs/remotes/origin/topicA/*
  fetch = refs/heads/topicB/*:refs/remotes/origin/topicB/*

> If a single repository has 1000 branches, that is a sign that
> there is something wrong in the project's organization.  No
> single person should need to look at 1000 branches -- that goes
> directly against the distributed nature of git.  Humans do not
> scale well.

Without saying how those 1000 branches are organised, I think that it is an 
incorrect generalisation to say that something is wrong.  I have tens of 
thousands of files in my home directory.  I cannot possibly look at all those 
files; and yet it's not disorganised.

To me, one of git's strongest features is the ability to have hierarchically 
organised branch names.  It solves the scalability problem in one fell swoop.

> But you cannot optimize humans the same way.

No; but you can optimise them in a different way.  Trees are the way, as long 
as the number of choices at any level is within reasonable limits, then you 
can sort through thousands and thousands of items easily.

I actually think the way to go with this globbing stuff is to make it possible 
to specify more accurately what you want.  The weak area at the moment is the 
fact that it's not possible to do a non-recursive glob.  That is to say that

 fetch = refs/heads/*:refs/remotes/*

would grab x/foo, y/bar, and z/baz.  I know you keep your topic branches in 
places like jc/some-topic and js/some-other-topic.  If you ever pushed them, 
then the default glob would pull them all.  If we could have a non-recursive 
glob, then you could safely push them (making things all the more 
distributed) without impacting the master, maint, next, etc branch pulls.

> Helping this kind of project organization is what git inherited
> from BK as well.  The key to scaling is to limit the product of
> the level of details times the size of the area one person needs

All agreed.  However, limiting the detail is not necessarily the same as 
hiding the detail.  If those 1000 branches can be arranged in such a manner 
as to make 990 of them easily ignored, then that's just as good as hiding 
those 990.



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
andyparkins@gmail.com

Re: [PATCH] git-remote exclude

[Johannes Schindelin]

Hi,

On Tue, 16 Jan 2007, Andy Parkins wrote:

> On Monday 2007 January 15 21:56, Junio C Hamano wrote:
> 
> > I do not think you are interested in "not these two but everything 
> > else".  You are interested in maint, master, next, pu and man.  You 
> > can get away by saying "not these two but everything else" only 
> > because you are implicitly trusting me not to publish insane number of 
> > random throw-away branches left and right.
> 
> Nope.  I think I'm right.  You're right that I'm trusting you not to 
> publish an insane number of branches, but that holds for the glob-mode 
> only, regardless of whether the NOT fetches are in place.  The sort of 
> thing I imagine happening is that as maintainer you suddenly say - I 
> think 1.5.0 is so good that we'll maintain it for longer than usual.  
> In which case we would suddenly get a branch for maint-1.5.0.  I would 
> want that branch, but I still don't want man, html and todo.

Fine. Submit a patch. And if that patch makes git-fetch not as fragile as 
I expect it to do, I am not opposed to inclusion. I'll just not use that 
feature. And I'll still find your usage confused.

Ciao,
Dscho

Re: [PATCH] git-remote exclude

[Quy Tonthat]

Johannes Schindelin wrote:
> Fine. Submit a patch. And if that patch makes git-fetch not as fragile as 
> I expect it to do, I am not opposed to inclusion. I'll just not use that 
> feature. And I'll still find your usage confused.
> 
I have no doubt your opposition to inclusion is important here. But some people
would (out-of-fashion'ly/arrogantly/dont-care'ly) submit their patches
"just for fun", as they did in the olden days. (Not that old to me though).

Quy

Re: [PATCH] git-remote exclude

[Johannes Schindelin]

Hi,

On Thu, 18 Jan 2007, Quy Tonthat wrote:

> Johannes Schindelin wrote:
> > Fine. Submit a patch. And if that patch makes git-fetch not as fragile as 
> > I expect it to do, I am not opposed to inclusion. I'll just not use that 
> > feature. And I'll still find your usage confused.
> > 
> I have no doubt your opposition to inclusion is important here. But some people
> would (out-of-fashion'ly/arrogantly/dont-care'ly) submit their patches
> "just for fun", as they did in the olden days. (Not that old to me though).

Hey, if others need that feature, I'm fine! And maybe I find myself in the 
need of exactly the same feature that I was opposed to, later in my life. 
Happened before, will happen again.

Ciao,
Dscho

Re: [PATCH] git-remote exclude

[Quy Tonthat]

Johannes Schindelin wrote:
> Hi,
> 
> On Mon, 15 Jan 2007, Quy Tonthat wrote:
> 
>> "git-remote exclude" can be used to prevent one or more unwanted remote 
>> branches from being tracked. After, for example,
>>     $ git-remote origin exclude man html
>> "git-fetch origin" will no longer fetch origin/man and origin/html.
> 
> That is not what your patch does.
> 
> It rewrites the "remote.$name.fetch" entries so that those branches are 
> not _updated_, but they are _fetched_ nevertheless.

I meant to say "track", but somehow "fetch" got to my fingers. Old ages, perhaps.
You are right, "update" is _the_ word. Thanks.

> But then, I don't really see _why_ you would want such a solution.
> After all, you are more likely to be interested in _specific_ branches, rather 
> than all branches _except_ a few.

For different situations, there are different paths to choose to reach that
ultimate "After All" (Zen? NO!). I offered one (little) path and expect to see
more of bigger ones. That's the _why_ (and/or, the _why_ not).

Quy