Users of git-check-files?

Users of git-check-files?

[Johannes Schindelin]

Hi,

there's git-check-files in the repository, but AFAIK nobody uses it, not 
even "git status", which would be the primary candidate. If really no 
users of git-check-files exist, maybe we should remove it?

Ciao,
Dscho

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Johannes Schindelin wrote:
> 
> there's git-check-files in the repository, but AFAIK nobody uses it, not 
> even "git status", which would be the primary candidate. If really no 
> users of git-check-files exist, maybe we should remove it?

Yes.

It was used by the old "git-applymbox" (aka "dotest") until I got around 
writing "git-apply".

It has no point any more, all the tools check the file status on their 
own, and yes, the thing should probably be removed.

		Linus

Re: Users of git-check-files?

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> It has no point any more, all the tools check the file status on their 
> own, and yes, the thing should probably be removed.

How about git-rev-tree?  Does anybody care?

Re: Users of git-check-files?

[Johannes Schindelin]

Hi,

On Tue, 2 Aug 2005, Junio C Hamano wrote:

> Linus Torvalds <torvalds@osdl.org> writes:
>
>> It has no point any more, all the tools check the file status on their
>> own, and yes, the thing should probably be removed.
>
> How about git-rev-tree?  Does anybody care?

I try to write a "git annotate" based on the output of git-whatchanged. 
Since git-whatchanged only shows the commit and its parent, I need to 
figure out what was the last diff that led to the current commit's state. 
This needs git-rev-tree.

Ciao,
Dscho

P.S.: My only unsolved problem is that git-whatchanged sometimes shows
the diffs in the wrong order (clock-skew problem?)

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Johannes Schindelin wrote:
> 
> I try to write a "git annotate" based on the output of git-whatchanged. 

You can't. It's fundamentally not doable, because you lose the merge 
information.

So you need to use a combination of git-rev-list _and_ git-whatchanged.

Use the "--parents" flag to git-rev-list to get the parents output for 
doing your own graph, if you want to.

I have been thinking of adding a "follow this file" mode to git-rev-list,
which just ignores all children of merges that used the file directly from
one of the other children. Exactly because then you could have a
git-rev-list that prunes based on filename, and would potentially be a lot
more efficient (not follow the "uninteresting" side of a merge).

I haven't gotten around to it, partly because I wonder if it would be
better as a separate program - even if that would mean that you'd lose all
the cool features of git-rev-list (the ability to limit it to certain
versions, and the ability to sort the output in relevant orders).

Partly because I'm just too friggin lazy.

> P.S.: My only unsolved problem is that git-whatchanged sometimes shows
> the diffs in the wrong order (clock-skew problem?)

Nope, this is a direct result of two branches modifying the same file in 
parallel. There is no "right" or "wrong" order.

		Linus

Re: Users of git-check-files?

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2354 bytes --]

Hi,

On Wed, 3 Aug 2005, Linus Torvalds wrote:

> On Wed, 3 Aug 2005, Johannes Schindelin wrote:
>>
>> I try to write a "git annotate" based on the output of git-whatchanged.
>
> You can't. It's fundamentally not doable, because you lose the merge
> information.

That's why I said I need git-rev-tree. In the meantime I discovered, that 
git-rev-list has a "--parents" option which suits me just fine. Therefore 
I'd say: kill git-rev-tree.

> So you need to use a combination of git-rev-list _and_ git-whatchanged.

I tried to do without the ugly "script is calling script is calling 
program" idiom. That is why my attempt at git-annotate (see attached) is 
so slow.

> I have been thinking of adding a "follow this file" mode to git-rev-list,
> which just ignores all children of merges that used the file directly from
> one of the other children. Exactly because then you could have a
> git-rev-list that prunes based on filename, and would potentially be a lot
> more efficient (not follow the "uninteresting" side of a merge).

Let's sit and see if people pick it up at all. If yes, I'd rather rewrite 
the whole thing in C eventually (it is in perl now and uses hashes quite 
extensively...).

>> P.S.: My only unsolved problem is that git-whatchanged sometimes shows
>> the diffs in the wrong order (clock-skew problem?)
>
> Nope, this is a direct result of two branches modifying the same file in
> parallel. There is no "right" or "wrong" order.

Exactly. But there is a "you probably meant that": the branch in which it 
was modified last (not counting merges, of course).

Notes:

- You can either annotate by commit (this is the default), or show 
some other informations with the "-f" flag: Try "-f author,commit:8".

- If you don't specify any files, it assumes you mean all files 
(Attention: slow).

- You can start at a commit instead of the current state by specifying
"-c other_commit".

- The list of commits is traversed as output by git-rev-list, i.e. 
chronologically. Each line is marked with the commit whose parent does not 
contain that line.

- I am not at all sure if my handling of merges is sane. The logic is like 
this: If the commit is parent to more than one commit (i.e. a merge), then 
the touched lines are tentatively marked as changed in that commit, but 
are possibly overridden at a later stage.

Ciao,
Dscho

[-- Attachment #2: Type: TEXT/PLAIN, Size: 8032 bytes --]

#!/usr/bin/perl

use Getopt::Std;

sub usage() {
	print STDERR 'Usage: ${\basename $0} [-s] [-f format] [-c commit] [files...]

	-s		only look at first parent in case of a merge
	-f format	revision format (e.g. "author,commit:8")
	-c commit	start looking at this commit
';

	exit(1);
}

getopts("hsf:c:") or usage();
$opt_h && usage();

$first_parent_only=$opt_s;

sub read_file ($) {
	my $file=$_[0];
	open IN, $file || return 1;
	$orig_line_count=0;
	@lines=();
	@line_handled=();
	@revisions=();
	while(<IN>) {
		$orig_line_count++;
		$lines[$orig_line_count]=$_;
	}
	close IN;
	$orig_line_count>0 || return 2;
	@mapping[1..$orig_line_count]=(1..$orig_line_count);
	return 0;
}
	
sub init_file($$) {
	my $file=$_[0];
	my $head=$_[1];

	if($head eq "") {
		# read current file
		my $ret=read_file("<".$file);
		$ret==0 || return $ret;

		$current_revision="*"x($revision_string_length-1).";";
		handle_diff("git-diff-files -p ".$file."|",1);
	} else {
		if(`git-ls-tree $head $file`=~/^\S+\s+\S+\s+(\S+)/) {
			$sha1=$1;
			my $ret=read_file("git-cat-file blob ".$sha1."|");
			$ret==0 || return $ret;
		} else {
			usage();
		}
	}
	$file_version=0;
	return 0;
}

# mark all lines still unaccounted for
sub mark_all ($) {
	my $mark_lines_as_handled=$_[0];
	foreach $line (@mapping) {
		if($line_handled{$line}==undef) {
			$revisions[$line]=get_revision();
			if($mark_lines_as_handled) {
				$line_handled{$line}=1;
			}
		}
	}
	if($mark_lines_as_handled) {
		$orig_line_count=0;
		@mapping=();
	}
}

# this sub only handles unified diffs
sub handle_diff($$) {
	my $diff=$_[0];
	my $mark_lines_as_handled=$_[1];

	open DIFF, $diff;
	my @new_mapping=();
	my $current_line_nr_minus=1;
	my $current_line_nr_plus=1;

	while(<DIFF>) {
		if(/^@@ -(\d+),(\d+) \+(\d+),(\d+) @@/) {
			$empty_diff=0;
			$start_minus=$1;
			$count_minus=$2;
			$start_plus=$3;
			$count_plus=$4;

			# if file was created here, were finished
			if($start_minus==0) {
				mark_all($mark_lines_as_handled);
				return;
			}
			# sane check
			$start_minus-$current_line_nr_minus==$start_plus-$current_line_nr_plus
				|| die "invalid diff: $start_minus,$current_line_nr_minus,$start_plus,$current_line_nr_plus";
			if($start_minus-$current_line_nr_minus>0) {
				@new_mapping[$current_line_nr_minus..$start_minus]
					=@mapping[$current_line_nr_plus..$start_plus];
				$current_line_nr_minus=$start_minus;
				$current_line_nr_plus=$start_plus;
			}
			while($count_minus>0 || $count_plus>0) {
				$_=<DIFF>;
				if(/^-/) {
					$new_mapping[$current_line_nr_minus]=undef;
					$current_line_nr_minus++;
					$count_minus--;
				} else {
					if(/^\+/) {
						$orig=$mapping[$current_line_nr_plus];
						if($orig>0 && $line_handled[$orig]==undef) {
							$revisions[$orig]=get_revision();
							if($mark_lines_as_handled) {
								$line_handled[$orig]=1;
								$orig_line_count--;
								if($orig_line_count==0) {
									@mapping=@new_mapping;
									return;
								}
							}
						}
						$current_line_nr_plus++;
						$count_plus--;
					} else {
						$orig_line_nr=$mapping[$current_line_nr_plus];
						$new_mapping[$current_line_nr_minus]=$orig_line_nr;
						if($orig_line_nr>0) {
							# sane check
							if(substr($_,1) ne $lines[$orig_line_nr]) {
								print "--\n";
								print @lines[($orig_line_nr-3)..($orig_line_nr+3)];
								print "--\n";
								die "invalid diff ($orig_line_nr:$current_line_nr_plus): ".substr($_,1)." is not ".$lines[$orig_line_nr];
							}
							substr($_,1) eq $lines[$orig_line_nr] ||
								die "invalid diff ($diff): ".substr($_,1)." is not ".$lines[$orig_line_nr].$lines[1..$#lines];
						}
						$current_line_nr_minus++;
						$current_line_nr_plus++;
						$count_minus--;
						$count_plus--;
					}
				}
			}
		}
	}
	close DIFF;
	$rest_lines=$#mapping-$current_line_nr_plus;
	if($rest_lines>0) {
		@new_mapping[$current_line_nr_minus..($current_line_nr_minus+$rest_lines)]
			=@mapping[$current_line_nr_plus..($current_line_nr_plus+$rest_lines)];
	}
	$file_version++;
	@mapping=@new_mapping;
	return;
}

sub get_revision() {
	if($current_revision eq "") {
		%commit_values=('fileversion','V-'.$file_version);
		$commit_values{'commit'}=$current_commit;
		open COMMIT, "git-cat-file commit $current_commit|";
		while(($_=<COMMIT>) && !/^$/) {
			if(/^parent ([0-9a-f]{40})/) {
				$commit_values{'parents'}.=$1." ";
			} elsif(/^author (.*)/) {
				$commit_values{'author'}=$1;
			} elsif(/^committer (.*)/) {
				$commit_values{'committer'}=$1;
			}
		}
		close COMMIT;

		for($i=0;$i<$#revision_format;$i+=2) {
			my $temp=$commit_values{$revision_format[$i]};
			my $length=length($temp);
			if($length>$revision_format[$i+1]) {
				$temp=substr($temp,0,$revision_format[$i+1]);
			} elsif($length<$revision_format[$i+1]) {
				$temp.=" "x($revision_format[$i+1]-$length);
			}
			$current_revision.=$temp.";";
		}
	}
	return $current_revision;
}

sub show_current {
	for($i=1;$i<=$#lines;$i++) {
		print $revisions[$i].$lines[$i];
	}
}

sub annotate_file($$) {
	my $file=$_[0];
	my $head=$_[1];

	init_file($file,$head);

	my @new_mapping=@mapping;
	%commit_mappings=($commits[0]=>\@new_mapping);

	my $index;
	for($index=0;$index<$#commits;$index++) {
		$current_commit=$commits[$index];
		$current_revision="";
		if($orig_line_count==0 || $children_count{$current_commit}==0) {
			mark_all(1);
			show_current();
			return;
		} else {
			$next_commit=$commits[$index+1];
			if($parent_tree{$next_commit} ne $current_commit) {
				# current_commit has children, so save the line mapping
				my @new_mapping=@mapping; # force copy
				$commit_mappings{$current_commit}=\@new_mapping;

				# get current_commit (which is the parent of next_commit)
				$current_commit=$parent_tree{$next_commit};
				if($commit_mappings{$current_commit}==undef) {
					die "fatal";
				}
				@mapping=@{$commit_mappings{$current_commit}};

				# free memory
				if($children_count{$current_commit}==1) {
					$commit_mappings{$current_commit}=undef; 
				}
			}
			handle_diff("git-diff-tree -p $next_commit $current_commit $file|",$children_count{$current_commit}==1);
			# if there are multiple children, save the line mapping
			if($children_count{$next_commit}>1) {
				my @new_mapping=@mapping; # force copy
				$commit_mappings{$next_commit}=\@new_mapping;
			}
		}
	}
	show_current();
}

sub parse_revision_format($) {
	my $format=$_[0];
	$revision_string_length=0;
	@revision_format=();
	foreach $f (split(",",$format)) {
		if($f=~/^(.*):(.*)/) {
			push @revision_format,($1,$2);
		} else {
			push @revision_format,($f,16);
		}
		$revision_string_length+=$revision_format[$#revision_format]+1;
	}
	if($#revision_format<1) {
		push @revision_format,("commit",40);
		$revision_string_length=$revision_format[$#revision_format]+1;
	}
}

sub get_parent_tree($) {
	my $head=$_[0];
	my $i;

	%parent_tree=();
	%children_count=();
	@commits=();
	$next_commit="";

	open REVS, "git-rev-list --parents ".($head eq ""?"HEAD":$head)."|";
	LOOP: while(<REVS>) {
		if($first_parent_only && $next_commit ne "") {
			while(!/^$next_commit/) {
				if(!($_=<REVS>)) {
					last LOOP;
				}
			}
		}
		my @list=split /[ \n]/;
		push @commits, $list[0];
		for($i=1;$i<=$#list;$i++) {
			$parent_tree{$list[$i]}=$list[0];
		}
		$children_count{$list[0]}=$#list;
		if($first_parent_only) {
			$next_commit=$list[1];
		}
	}
	close REVS;
}

get_parent_tree($opt_c);
parse_revision_format($opt_f);

if($#ARGV<0) {
	open FILES, 'git-ls-tree -r '.($opt_c eq ''?'HEAD':$opt_c).'|';
	while(<FILES>) {
		if(/^\S+\s+\S+\s+\S+\s+(.*)$/) {
			push @ARGV, $1;
		}
	}
	close FILES;
}

for($i=0;$i<=$#ARGV;$i++) {
	if($#ARGV>1) {
		print "File: ".$ARGV[$i]."\n";
	}
	annotate_file($ARGV[$i],$opt_c);
}

Re: Users of git-check-files?

[Linus Torvalds]

On Tue, 2 Aug 2005, Junio C Hamano wrote:
> 
> How about git-rev-tree?  Does anybody care?

Yeah, probably not. git-rev-list does so much more than git-rev-tree ever 
did.

		Linus

Re: Users of git-check-files?

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> Yeah, probably not. git-rev-list does so much more than git-rev-tree ever 
> did.

Does rev-list do --edges ;-)?

BTW, I have two known bugs/problems that I haven't resolved,
which is bothering me quite a bit.  Yes, it is my fault (lack of
time and concentration).  One is the EINTR from the daemon.c,
and another is receive-pack failing with "unpack should have
generated but I cannot find it" when pushing.  The latter is
something "I am aware of, I know it needs to be diagnosed, but I
have not looked into yet".

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Junio C Hamano wrote:
>
> Linus Torvalds <torvalds@osdl.org> writes:
> 
> > Yeah, probably not. git-rev-list does so much more than git-rev-tree ever 
> > did.
> 
> Does rev-list do --edges ;-)?

No, but does anybody use it? It _may_ be interesting as a git-merge-base
thing, but then we should probably add it to git-merge-base (which
actually _does_ do edges these days, but it just only shows the most
recent one..)

> BTW, I have two known bugs/problems that I haven't resolved,
> which is bothering me quite a bit.  Yes, it is my fault (lack of
> time and concentration).  One is the EINTR from the daemon.c,

I actually think the new code is totally broken. If you want to listen to 
several sockets, you should just run several daemons. Yeah, yeah, it won't 
give you "global child control" etc, but I doubt we care.

But I don't think it's a huge issue, since most serious users are likely 
to use the "--inetd" flag and use inetd as the real daemon anyway (ie the 
built-in daemon code is most useful for normal users just setting up 
their own quick thing).

> and another is receive-pack failing with "unpack should have
> generated but I cannot find it" when pushing.  The latter is
> something "I am aware of, I know it needs to be diagnosed, but I
> have not looked into yet".

I've lost that state. Can you explain a bit mroe..

		Linus

Re: Users of git-check-files?

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> I've lost that state. Can you explain a bit mroe..

Sorry, you have not lost anything.  It is my bad that this is
the first time I brought it up.  I've been seeing that from time
to time when I push to either my "send to master" repository
from my working repository, or from the "send to master"
repository to master.kernel.org, but I haven't figured it out if
there is any pattern.  It's one of those "I'll try to take a
snapshot so I can have a reproduction recipe to figure it out
next time it happens" things.  Unfortunately, when it happens
against master.kernel.org, I have more urgent task of making
sure that the public repository is not in any way corrupted, and
after fixing that (it typically takes removing the
git.git/refs/{master,pu} and repushing, which recovers fine), I
do not have much to start trying to reproduce it anymore X-<.

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Junio C Hamano wrote:
>
> Linus Torvalds <torvalds@osdl.org> writes:
> 
> > I've lost that state. Can you explain a bit mroe..
> 
> Sorry, you have not lost anything.  It is my bad that this is
> the first time I brought it up.  I've been seeing that from time
> to time when I push to either my "send to master" repository
> from my working repository, or from the "send to master"
> repository to master.kernel.org, but I haven't figured it out if
> there is any pattern.

Are you sure you have a good git version on master? I've never seen 
anything like that, and I push all the time..

		Linus

Re: Users of git-check-files?

[Johannes Schindelin]

Hi,

On Wed, 3 Aug 2005, Linus Torvalds wrote:

> Are you sure you have a good git version on master? I've never seen
> anything like that, and I push all the time..

Call him Zaphod: he has two heads (master and pu). You don't. As I said in 
another mail, this could be very well related to Junio's problems.

Ciao,
Dscho

Re: Users of git-check-files?

[Josef Weidendorfer]

On Wednesday 03 August 2005 18:50, you wrote:
> Hi,
>
> On Wed, 3 Aug 2005, Linus Torvalds wrote:
> > Are you sure you have a good git version on master? I've never seen
> > anything like that, and I push all the time..
>
> Call him Zaphod: he has two heads (master and pu). You don't. As I said in
> another mail, this could be very well related to Junio's problems.

Yes it is. To reproduce:
Create a repository with 2 branches.
Make 2 clones of the 2 branches via SSH.
Make a commit on one clone and push.
Make another commit on the other clone and push => ERROR

A log of this last push:
=============================================
~/tmp/git/clone2> cg-push
'refs/heads/branch2': updating from 80e4d426dd4c865b943cc1121b580a946eee921d 
to 8196067677e3415ce404ea5bc35731ac7d56115d
fatal: bad object f7e944b036fd00af656b262140c1dc93ceffadb1
Packing 0 objects
Unpacking 0 objects

fatal: unpack should have generated 8196067677e3415ce404ea5bc35731ac7d56115d, 
but I can't find it!
=============================================

f7e9... is the commit pushed from the first clone.

I had the same problem yesterday.

Josef

Re: Users of git-check-files?

[Junio C Hamano]

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

> ~/tmp/git/clone2> cg-push
> 'refs/heads/branch2': updating from 80e4d426dd4c865b943cc1121b580a946eee921d 
> to 8196067677e3415ce404ea5bc35731ac7d56115d
> fatal: bad object f7e944b036fd00af656b262140c1dc93ceffadb1
> Packing 0 objects
> Unpacking 0 objects

Ahhhh.

Yes, we read from the remote and use old->sha1 when doing the
pack generation.  If we do not have that object (in Josef's
example because it is not something we have pulled yet), then
rev-list has no way to create the necessary pack.

I have a feeling this explains what I've been seeing as well.
The first thing I do before I push is to fetch (but not
merge-pull) from kernel.org, but this happens against
www.kernel.org (to make sure people are seeing what I want them
to see), not against master.  So if I happen to do my push cycle
twice before the things propagate from master to www.kernel.org,
_and_ if I happened to have run git prune in my private
repository just before the second cycle, I may end up not having
the objects referenced by "pu" branch there in my private
repository.

Re: Users of git-check-files?

[Josef Weidendorfer]

On Wednesday 03 August 2005 19:08, you wrote:
> Yes it is. To reproduce:

You do not need 2 clones.
It is enough to have one clone with a branch, and you make a commit in the 
original repository.
Afterwards, pushing a new commit from the clone gives the error.

After pulling the missing commit from the original rep, the push works.

Josef

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Josef Weidendorfer wrote:
> 
> Yes it is. To reproduce:
> Create a repository with 2 branches.
> Make 2 clones of the 2 branches via SSH.
> Make a commit on one clone and push.
> Make another commit on the other clone and push => ERROR

This works perfectly fine, you just have to make sure that you update the 
right head.

If you try to update a head that is ahead of you, that is driver error. 
Admittedly one that could have nicer error messages ;)

This is why git-send-pack takes the name of the branch to update..

The real problem with git-send-pack is that the local and remote names 
have to be the same, which is a bug, really. It _should_ be perfectly fine 
to do something like

	git-send-pack ..dest.. localname:remotename

which would push the local "localname" branch to the remote "remotename" 
branch.

			Linus

[PATCH] (preview) Renaming push.

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> The real problem with git-send-pack is that the local and remote names 
> have to be the same, which is a bug, really. It _should_ be perfectly fine 
> to do something like
>
> 	git-send-pack ..dest.. localname:remotename
>
> which would push the local "localname" branch to the remote "remotename" 
> branch.

Yes, and that is what I have been cooking today ;-).

This patch is a preview, only because I have not converted the
pull side yet.  My limited testing shows that you can push refs
under different names fine with this patch.

The plan is to also let people say the renaming for pull side
like this.

        git-fetch-pack <remote> remotename:localname...

Note that remote/local is confusing when we consider both pull
and push sides, so we should call them srcname and dstname.  The
commit message uses these words.

-jc

------------
This allows git-send-pack to push local refs to a destination
repository under different names.

Here is the name mapping rules for refs.

* If there is no ref mapping on the command line:

 - if '--all' is specified, it is equivalent to specifying
   <local> ":" <local> for all the existing local refs on the
   command line
 - otherwise, it is equivalent to specifying <ref> ":" <ref> for
   all the refs that exist on both sides.

* <name> is just a shorthand for <name> ":" <name>

* <src> ":" <dst>

  push ref that matches <src> to ref that matches <dst>.

  - It is an error if <src> does not match exactly one of local
    refs.

  - It is an error if <dst> matches more than one remote refs.

  - If <dst> does not match any remote refs, either

    - it has to start with "refs/"; <dst> is used as the
      destination literally in this case.

    - <src> == <dst> and the ref that matched the <src> must not
      exist in the set of remote refs; the ref matched <src>
      locally is used as the name of the destination.

For example,

  - "git-send-pack --all <remote>" works exactly as before;

  - "git-send-pack <remote> master:upstream" pushes local master
    to remote ref that matches "upstream".  If there is no such
    ref, it is an error.

  - "git-send-pack <remote> master:refs/heads/upstream" pushes
    local master to remote refs/heads/upstream, even when
    refs/heads/upstream does not exist.

  - "git-send-pack <remote> master" into an empty remote
    repository pushes the local ref/heads/master to the remote
    ref/heads/master.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 cache.h     |    3 +
 connect.c   |  172 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 send-pack.c |  146 ++++++++++++++++++++------------------------------
 3 files changed, 230 insertions(+), 91 deletions(-)

b959d297c7206b0b634160ac015122f70389415b
diff --git a/cache.h b/cache.h
--- a/cache.h
+++ b/cache.h
@@ -298,12 +298,15 @@ struct ref {
 	struct ref *next;
 	unsigned char old_sha1[20];
 	unsigned char new_sha1[20];
+	struct ref *peer_ref; /* when renaming */
 	char name[0];
 };
 
 extern int git_connect(int fd[2], char *url, const char *prog);
 extern int finish_connect(pid_t pid);
 extern int path_match(const char *path, int nr, char **match);
+extern int match_refs(struct ref *src, struct ref *dst, struct ref ***dst_tail,
+		      int nr_refspec, char **refspec, int all);
 extern int get_ack(int fd, unsigned char *result_sha1);
 extern struct ref **get_remote_heads(int in, struct ref **list, int nr_match, char **match);
 
diff --git a/connect.c b/connect.c
--- a/connect.c
+++ b/connect.c
@@ -31,11 +31,9 @@ struct ref **get_remote_heads(int in, st
 		name = buffer + 41;
 		if (nr_match && !path_match(name, nr_match, match))
 			continue;
-		ref = xmalloc(sizeof(*ref) + len - 40);
+		ref = xcalloc(1, sizeof(*ref) + len - 40);
 		memcpy(ref->old_sha1, old_sha1, 20);
-		memset(ref->new_sha1, 0, 20);
 		memcpy(ref->name, buffer + 41, len - 40);
-		ref->next = NULL;
 		*list = ref;
 		list = &ref->next;
 	}
@@ -81,6 +79,174 @@ int path_match(const char *path, int nr,
 	return 0;
 }
 
+struct refspec {
+	char *src;
+	char *dst;
+};
+
+static struct refspec *parse_ref_spec(int nr_refspec, char **refspec)
+{
+	int i;
+	struct refspec *rs = xmalloc(sizeof(*rs) * (nr_refspec + 1));
+	for (i = 0; i < nr_refspec; i++) {
+		char *sp, *dp, *ep;
+		sp = refspec[i];
+		ep = strchr(sp, ':');
+		if (ep) {
+			dp = ep + 1;
+			*ep = 0;
+		}
+		else
+			dp = sp;
+		rs[i].src = sp;
+		rs[i].dst = dp;
+	}
+	rs[nr_refspec].src = rs[nr_refspec].dst = NULL;
+	return rs;
+}
+
+static int count_refspec_match(const char *pattern,
+			       struct ref *refs,
+			       struct ref **matched_ref)
+{
+	int match;
+	int patlen = strlen(pattern);
+
+	for (match = 0; refs; refs = refs->next) {
+		char *name = refs->name;
+		int namelen = strlen(name);
+		if (namelen < patlen ||
+		    memcmp(name + namelen - patlen, pattern, patlen))
+			continue;
+		if (namelen != patlen && name[namelen - patlen - 1] != '/')
+			continue;
+		match++;
+		*matched_ref = refs;
+	}
+	return match;
+}
+
+static void link_dst_tail(struct ref *ref, struct ref ***tail)
+{
+	**tail = ref;
+	*tail = &ref->next;
+	**tail = NULL;
+}
+
+static int match_explicit_refs(struct ref *src, struct ref *dst,
+			       struct ref ***dst_tail, struct refspec *rs)
+{
+	int i, errs;
+	for (i = errs = 0; rs[i].src; i++) {
+		struct ref *matched_src, *matched_dst;
+
+		matched_src = matched_dst = NULL;
+		switch (count_refspec_match(rs[i].src, src, &matched_src)) {
+		case 1:
+			break;
+		case 0:
+			errs = 1;
+			error("src refspec %s does not match any.");
+			break;
+		default:
+			errs = 1;
+			error("src refspec %s matches more than one.",
+			      rs[i].src);
+			break;
+		}
+		switch (count_refspec_match(rs[i].dst, dst, &matched_dst)) {
+		case 1:
+			break;
+		case 0:
+			if (!memcmp(rs[i].dst, "refs/", 5)) {
+				int len = strlen(rs[i].dst) + 1;
+				matched_dst = xcalloc(1, sizeof(*dst) + len);
+				memcpy(matched_dst->name, rs[i].dst, len);
+				link_dst_tail(matched_dst, dst_tail);
+			}
+			else if (!strcmp(rs[i].src, rs[i].dst) &&
+				 matched_src) {
+				/* pushing "master:master" when
+				 * remote does not have master yet.
+				 */
+				int len = strlen(matched_src->name);
+				matched_dst = xcalloc(1, sizeof(*dst) + len);
+				memcpy(matched_dst->name, matched_src->name,
+				       len);
+				link_dst_tail(matched_dst, dst_tail);
+			}
+			else {
+				errs = 1;
+				error("dst refspec %s does not match any "
+				      "existing ref on the remote and does "
+				      "not start with refs/.", rs[i].dst);
+			}
+			break;
+		default:
+			errs = 1;
+			error("dst refspec %s matches more than one.",
+			      rs[i].dst);
+			break;
+		}
+		if (errs)
+			continue;
+		if (matched_src->peer_ref) {
+			errs = 1;
+			error("src ref %s is sent to more than one dst.",
+			      matched_src->name);
+		}
+		else
+			matched_src->peer_ref = matched_dst;
+		if (matched_dst->peer_ref) {
+			errs = 1;
+			error("dst ref %s receives from more than one src.",
+			      matched_dst->name);
+		}
+		else
+			matched_dst->peer_ref = matched_src;
+	}
+	return -errs;
+}
+
+static struct ref *find_ref_by_name(struct ref *list, const char *name)
+{
+	for ( ; list; list = list->next)
+		if (!strcmp(list->name, name))
+			return list;
+	return NULL;
+}
+
+int match_refs(struct ref *src, struct ref *dst, struct ref ***dst_tail,
+	       int nr_refspec, char **refspec, int all)
+{
+	struct refspec *rs = parse_ref_spec(nr_refspec, refspec);
+
+	if (nr_refspec)
+		return match_explicit_refs(src, dst, dst_tail, rs);
+
+	/* pick the remainder */
+	for ( ; src; src = src->next) {
+		struct ref *dst_peer;
+		if (src->peer_ref)
+			continue;
+		dst_peer = find_ref_by_name(dst, src->name);
+		if (dst_peer && dst_peer->peer_ref)
+			continue;
+		if (!dst_peer) {
+			if (!all)
+				continue;
+			/* Create a new one and link it */
+			int len = strlen(src->name) + 1;
+			dst_peer = xcalloc(1, sizeof(*dst_peer) + len);
+			memcpy(dst_peer->name, src->name, len);
+			memcpy(dst_peer->new_sha1, src->new_sha1, 20);
+			link_dst_tail(dst_peer, dst_tail);
+		}
+		dst_peer->peer_ref = src;
+	}
+	return 0;
+}
+
 enum protocol {
 	PROTO_LOCAL = 1,
 	PROTO_SSH,
diff --git a/send-pack.c b/send-pack.c
--- a/send-pack.c
+++ b/send-pack.c
@@ -43,7 +43,8 @@ static void exec_rev_list(struct ref *re
 		char *buf = malloc(100);
 		if (i > 900)
 			die("git-rev-list environment overflow");
-		if (!is_zero_sha1(refs->old_sha1)) {
+		if (!is_zero_sha1(refs->old_sha1) &&
+		    has_sha1_file(refs->old_sha1)) {
 			args[i++] = buf;
 			snprintf(buf, 50, "^%s", sha1_to_hex(refs->old_sha1));
 			buf += 50;
@@ -103,21 +104,6 @@ static int pack_objects(int fd, struct r
 	return 0;
 }
 
-static int read_ref(const char *ref, unsigned char *sha1)
-{
-	int fd, ret;
-	char buffer[60];
-
-	fd = open(git_path("%s", ref), O_RDONLY);
-	if (fd < 0)
-		return -1;
-	ret = -1;
-	if (read(fd, buffer, sizeof(buffer)) >= 40)
-		ret = get_sha1_hex(buffer, sha1);
-	close(fd);
-	return ret;
-}
-
 static int ref_newer(const unsigned char *new_sha1, const unsigned char *old_sha1)
 {
 	struct commit *new, *old;
@@ -143,108 +129,92 @@ static int ref_newer(const unsigned char
 	return 0;
 }
 
-static int local_ref_nr_match;
-static char **local_ref_match;
-static struct ref *local_ref_list;
-static struct ref **local_last_ref;
+static struct ref *local_refs, **local_tail;
+static struct ref *remote_refs, **remote_tail;
 
-static int try_to_match(const char *refname, const unsigned char *sha1)
+static int one_local_ref(const char *refname, const unsigned char *sha1)
 {
 	struct ref *ref;
-	int len;
-
-	if (!path_match(refname, local_ref_nr_match, local_ref_match)) {
-		if (!send_all)
-			return 0;
-
-		/* If we have it listed already, skip it */
-		for (ref = local_ref_list ; ref ; ref = ref->next) {
-			if (!strcmp(ref->name, refname))
-				return 0;
-		}
-	}
-
-	len = strlen(refname)+1;
-	ref = xmalloc(sizeof(*ref) + len);
-	memset(ref->old_sha1, 0, 20);
+	int len = strlen(refname) + 1;
+	ref = xcalloc(1, sizeof(*ref) + len);
 	memcpy(ref->new_sha1, sha1, 20);
 	memcpy(ref->name, refname, len);
-	ref->next = NULL;
-	*local_last_ref = ref;
-	local_last_ref = &ref->next;
+	*local_tail = ref;
+	local_tail = &ref->next;
 	return 0;
 }
 
-static int send_pack(int in, int out, int nr_match, char **match)
+static void get_local_heads(void)
+{
+	local_tail = &local_refs;
+	for_each_ref(one_local_ref);
+}
+
+static int send_pack(int in, int out, int nr_refspec, char **refspec)
 {
-	struct ref *ref_list, **last_ref;
 	struct ref *ref;
 	int new_refs;
 
-	/* First we get all heads, whether matching or not.. */
-	last_ref = get_remote_heads(in, &ref_list, 0, NULL);
-
+	/* No funny business with the matcher */
+	remote_tail = get_remote_heads(in, &remote_refs, 0, NULL);
+	get_local_heads();
+
+	/* match them up */
+	if (!remote_tail)
+		remote_tail = &remote_refs;
+	if (match_refs(local_refs, remote_refs, &remote_tail,
+		       nr_refspec, refspec, send_all))
+		return -1;
 	/*
-	 * Go through the refs, see if we want to update
-	 * any of them..
+	 * Finally, tell the other end!
 	 */
-	for (ref = ref_list; ref; ref = ref->next) {
-		unsigned char new_sha1[20];
-		char *name = ref->name;
-
-		if (nr_match && !path_match(name, nr_match, match))
-			continue;
-
-		if (read_ref(name, new_sha1) < 0)
-			continue;
-
-		if (!memcmp(ref->old_sha1, new_sha1, 20)) {
-			fprintf(stderr, "'%s' unchanged\n", name);
+	new_refs = 0;
+	for (ref = remote_refs; ref; ref = ref->next) {
+		char old_hex[60], *new_hex;
+		if (!ref->peer_ref)
 			continue;
+		if (!is_zero_sha1(ref->old_sha1)) {
+			if (!has_sha1_file(ref->old_sha1)) {
+				error("remote '%s' object %s does not "
+				      "exist on local",
+				      ref->name, sha1_to_hex(ref->old_sha1));
+				continue;
+			}
+			if (!ref_newer(ref->peer_ref->new_sha1,
+				       ref->old_sha1)) {
+				error("remote ref '%s' is not a strict "
+				      "subset of local ref '%s'.", ref->name,
+				      ref->peer_ref->name);
+				continue;
+			}
 		}
-
-		if (!ref_newer(new_sha1, ref->old_sha1)) {
-			error("remote '%s' isn't a strict parent of local", name);
+		if (!memcmp(ref->old_sha1, ref->peer_ref->new_sha1, 20)) {
+			fprintf(stderr, "'%s': up-to-date\n", ref->name);
 			continue;
 		}
-
-		/* Ok, mark it for update */
-		memcpy(ref->new_sha1, new_sha1, 20);
-	}
-
-	/*
-	 * See if we have any refs that the other end didn't have
-	 */
-	if (nr_match || send_all) {
-		local_ref_nr_match = nr_match;
-		local_ref_match = match;
-		local_ref_list = ref_list;
-		local_last_ref = last_ref;
-		for_each_ref(try_to_match);
-	}
-
-	/*
-	 * Finally, tell the other end!
-	 */
-	new_refs = 0;
-	for (ref = ref_list; ref; ref = ref->next) {
-		char old_hex[60], *new_hex;
-		if (is_zero_sha1(ref->new_sha1))
+		memcpy(ref->new_sha1, ref->peer_ref->new_sha1, 20);
+		if (is_zero_sha1(ref->new_sha1)) {
+			error("cannot happen anymore");
 			continue;
+		}
 		new_refs++;
 		strcpy(old_hex, sha1_to_hex(ref->old_sha1));
 		new_hex = sha1_to_hex(ref->new_sha1);
 		packet_write(out, "%s %s %s", old_hex, new_hex, ref->name);
-		fprintf(stderr, "'%s': updating from %s to %s\n", ref->name, old_hex, new_hex);
+		fprintf(stderr, "updating '%s'", ref->name);
+		if (strcmp(ref->name, ref->peer_ref->name))
+			fprintf(stderr, " using '%s'", ref->peer_ref->name);
+		fprintf(stderr, "\n  from %s\n  to   %s\n", old_hex, new_hex);
 	}
-	
+
 	packet_flush(out);
 	if (new_refs)
-		pack_objects(out, ref_list);
+		pack_objects(out, remote_refs);
 	close(out);
 	return 0;
 }
 
+
 int main(int argc, char **argv)
 {
 	int i, nr_heads = 0;

Re: [PATCH] (preview) Renaming push.

[Linus Torvalds]

On Wed, 3 Aug 2005, Junio C Hamano wrote:
>
> This allows git-send-pack to push local refs to a destination
> repository under different names.

Looks good, except I was almost hoping for one modification:

> Here is the name mapping rules for refs.
> 
> * If there is no ref mapping on the command line:
> 
>  - if '--all' is specified, it is equivalent to specifying
>    <local> ":" <local> for all the existing local refs on the
>    command line
>  - otherwise, it is equivalent to specifying <ref> ":" <ref> for
>    all the refs that exist on both sides.
> 
> * <name> is just a shorthand for <name> ":" <name>
> 
> * <src> ":" <dst>
> 
>   push ref that matches <src> to ref that matches <dst>.

In this format, "src" makes sense as a generic sha1_name, and _not_ 
necessarily limited to a ref-name.

IOW, there's really no reason to not allow

	git-send-pack .. dest .. <sha1>:dst-ref

where "<sha1>" may be something else than a ref (but a ref obviously ends 
up being one such thing).

For example, let's say that I've got stuff at the HEAD of my tree that I'm
not quite ready to push out yet, but I _do_ want to push out the parent of
the current head. I'd love to be able to just do

	git-send-pack parent $(git-rev-parse HEAD^):master

and there's no real reason why that syntax shouldn't just work: it's 
entirely logical to say "I want to push out the parent of my HEAD as 
'master' on the other end", and that's _exactly_ what the above says.

So:

>   - It is an error if <src> does not match exactly one of local
>     refs.

I think "src" should just be seen as any random SHA1 name, and we should 
use "get_sha()" to convert it to a SHA1.

In contrast, "dst" is obviously a real ref name, and as such:

>   - It is an error if <dst> matches more than one remote refs.

makes sense.

That makes your next rule a bit iffy, though:

>   - If <dst> does not match any remote refs, either
> 
>     - it has to start with "refs/"; <dst> is used as the
>       destination literally in this case.
> 
>     - <src> == <dst> and the ref that matched the <src> must not
>       exist in the set of remote refs; the ref matched <src>
>       locally is used as the name of the destination.

since "src" from a local standpoint isn't really a ref-name at all.

But hey, your current patch looks fine already. It doesn't _quite_ do what 
I had in mind, but it gets very very close.

		Linus

Re: [PATCH] (preview) Renaming push.

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> 	git-send-pack parent $(git-rev-parse HEAD^):master
>
> and there's no real reason why that syntax shouldn't just work: it's 
> entirely logical to say "I want to push out the parent of my HEAD as 
> 'master' on the other end", and that's _exactly_ what the above says.

Yes I think allowing get_sha()-able thing on the left hand side
makes things, well, interesting.  I need to think about it a bit.

When I do something like your example, I create a temporary
lightweight tag and push it.  Snapshots in JIT are just a bunch
of lightweight tags so..

Re: [PATCH] (preview) Renaming push.

[Linus Torvalds]

On Wed, 3 Aug 2005, Junio C Hamano wrote:
>
> > 	git-send-pack parent $(git-rev-parse HEAD^):master
> 
> When I do something like your example, I create a temporary
> lightweight tag and push it.  Snapshots in JIT are just a bunch
> of lightweight tags so..

I like the scripting, and combining pipelines of commands kind of thing. 

I agree that you can just make a temporary tag, but it's a bit like in any
scripting stuff - you could use a temp-file, but it's just cleaner if you
can keep temporary values in local variables (or in a local cmd pipeline
like the above).

Whenever you script somethign that creates a new tag or ref, you suddenly 
have cleanup and uniqueness issues etc. 

		Linus

Re: [PATCH] (preview) Renaming push.

[Junio C Hamano]

Well, I pushed it out, although I do agree that we should be
able to give anything get_sha()-able on the source side of the
push.  Probably a revised version should have the following
semantics:

    $ git-send-pack [--all] <remote> [<ref>...]

 - When no <ref> is specified:

   - with '--all', it is the same as specifying the full refs/*
     path for all local refs;

   - without '--all', it is the same as specifying the full
     refs/* path for refs that exist on both ends;

 - When one or more <ref>s are specified:

   - a single token <ref> (i.e. no colon) must be a pattern that
     tail-matches refs/* path for an existing local ref.  It is
     an error for the pattern to match no local ref, or more
     than one local refs.  The matching ref is pushed to the
     remote end under the same name.

   - <src>:<dst> can have different cases.  <src> is first tried
     as the tail-matching pattern for refs/* path.

     - If more than one matches are found, it is an error.

     - If one match is found, <dst> must either match no remote
       ref and start with "refs/", or match exactly one remote
       ref.  That remote ref is updated with the sha1 value
       obtained by the <src> sha1.

     - If no match is found, it is given to get_extended_sha1();
       it is an error if get_extended_sha1() does not find an
       object name.  If it succeeds, <dst> must either match
       no remote ref and start with "refs/" or match exactly
       one remote ref.  That remote ref is updated with the sha1
       value.

Re: [PATCH] (preview) Renaming push.

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> In this format, "src" makes sense as a generic sha1_name, and _not_ 
> necessarily limited to a ref-name.
>
> IOW, there's really no reason to not allow
>
> 	git-send-pack .. dest .. <sha1>:dst-ref
>
> where "<sha1>" may be something else than a ref (but a ref obviously ends 
> up being one such thing).
>
> For example, let's say that I've got stuff at the HEAD of my tree that I'm
> not quite ready to push out yet, but I _do_ want to push out the parent of
> the current head. I'd love to be able to just do
>
> 	git-send-pack parent $(git-rev-parse HEAD^):master
>
> and there's no real reason why that syntax shouldn't just work: it's 
> entirely logical to say "I want to push out the parent of my HEAD as 
> 'master' on the other end", and that's _exactly_ what the above says.

While I have not updated the send-pack <src>:<dst> syntax, I
added a horrible hack that some people may love to see.  This
removes the need to use git-rev-parse from many commands.

This is a preview.  I've run the standard test suite and it does
not seem to break anything, but I expect it will stay in "pu"
for a while.

------------
[PATCH] Update get_sha1() to grok extended format.

Everybody envies rev-parse, who is the only one that can grok
the extended sha1 format.  Move the get_extended_sha1() out of
rev-parse, rename it to get_sha1() and make it available to
everybody else.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 Makefile    |    3 +
 rev-parse.c |  188 +-------------------------------------------------
 sha1_file.c |   39 ----------
 sha1_name.c |  222 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 229 insertions(+), 223 deletions(-)
 create mode 100644 sha1_name.c

99ef7de3e234347d2af1ab2129ce99c5876b591b
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -95,7 +95,8 @@ LIB_H=cache.h object.h blob.h tree.h com
 	pack.h pkt-line.h refs.h
 LIB_OBJS=read-cache.o sha1_file.o usage.o object.o commit.o tree.o blob.o \
 	 tag.o date.o index.o diff-delta.o patch-delta.o entry.o path.o \
-	 refs.o csum-file.o pack-check.o pkt-line.o connect.o ident.o
+	 refs.o csum-file.o pack-check.o pkt-line.o connect.o ident.o \
+	 sha1_name.o
 
 LIB_H += rev-cache.h
 LIB_OBJS += rev-cache.o
diff --git a/rev-parse.c b/rev-parse.c
--- a/rev-parse.c
+++ b/rev-parse.c
@@ -21,8 +21,6 @@ static int output_sq = 0;
 #define REVERSED 1
 static int show_type = NORMAL;
 
-static int get_extended_sha1(char *name, unsigned char *sha1);
-
 /*
  * Some arguments are relevant "revision" arguments,
  * others are about output format or other details.
@@ -107,182 +105,6 @@ static void show_arg(char *arg)
 		show_norev(arg);
 }
 
-static int get_parent(char *name, unsigned char *result, int idx)
-{
-	unsigned char sha1[20];
-	int ret = get_extended_sha1(name, sha1);
-	struct commit *commit;
-	struct commit_list *p;
-
-	if (ret)
-		return ret;
-	commit = lookup_commit_reference(sha1);
-	if (!commit)
-		return -1;
-	if (parse_commit(commit))
-		return -1;
-	if (!idx) {
-		memcpy(result, commit->object.sha1, 20);
-		return 0;
-	}
-	p = commit->parents;
-	while (p) {
-		if (!--idx) {
-			memcpy(result, p->item->object.sha1, 20);
-			return 0;
-		}
-		p = p->next;
-	}
-	return -1;
-}
-
-static int find_short_object_filename(int len, const char *name, unsigned char *sha1)
-{
-	static char dirname[PATH_MAX];
-	char hex[40];
-	DIR *dir;
-	int found;
-
-	snprintf(dirname, sizeof(dirname), "%s/%.2s", get_object_directory(), name);
-	dir = opendir(dirname);
-	sprintf(hex, "%.2s", name);
-	found = 0;
-	if (dir) {
-		struct dirent *de;
-		while ((de = readdir(dir)) != NULL) {
-			if (strlen(de->d_name) != 38)
-				continue;
-			if (memcmp(de->d_name, name + 2, len-2))
-				continue;
-			memcpy(hex + 2, de->d_name, 38);
-			if (++found > 1)
-				break;
-		}
-		closedir(dir);
-	}
-	if (found == 1)
-		return get_sha1_hex(hex, sha1) == 0;
-	return 0;
-}
-
-static int match_sha(unsigned len, const unsigned char *a, const unsigned char *b)
-{
-	do {
-		if (*a != *b)
-			return 0;
-		a++;
-		b++;
-		len -= 2;
-	} while (len > 1);
-	if (len)
-		if ((*a ^ *b) & 0xf0)
-			return 0;
-	return 1;
-}
-
-static int find_short_packed_object(int len, const unsigned char *match, unsigned char *sha1)
-{
-	struct packed_git *p;
-
-	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
-		unsigned num = num_packed_objects(p);
-		unsigned first = 0, last = num;
-		while (first < last) {
-			unsigned mid = (first + last) / 2;
-			unsigned char now[20];
-			int cmp;
-
-			nth_packed_object_sha1(p, mid, now);
-			cmp = memcmp(match, now, 20);
-			if (!cmp) {
-				first = mid;
-				break;
-			}
-			if (cmp > 0) {
-				first = mid+1;
-				continue;
-			}
-			last = mid;
-		}
-		if (first < num) {
-			unsigned char now[20], next[20];
-			nth_packed_object_sha1(p, first, now);
-			if (match_sha(len, match, now)) {
-				if (nth_packed_object_sha1(p, first+1, next) || !match_sha(len, match, next)) {
-					memcpy(sha1, now, 20);
-					return 1;
-				}
-			}
-		}	
-	}
-	return 0;
-}
-
-static int get_short_sha1(char *name, unsigned char *sha1)
-{
-	int i;
-	char canonical[40];
-	unsigned char res[20];
-
-	memset(res, 0, 20);
-	memset(canonical, 'x', 40);
-	for (i = 0;;i++) {
-		unsigned char c = name[i];
-		unsigned char val;
-		if (!c || i > 40)
-			break;
-		if (c >= '0' && c <= '9')
-			val = c - '0';
-		else if (c >= 'a' && c <= 'f')
-			val = c - 'a' + 10;
-		else if (c >= 'A' && c <='F') {
-			val = c - 'A' + 10;
-			c -= 'A' - 'a';
-		}
-		else
-			return -1;
-		canonical[i] = c;
-		if (!(i & 1))
-			val <<= 4;
-		res[i >> 1] |= val;
-	}
-	if (i < 4)
-		return -1;
-	if (find_short_object_filename(i, canonical, sha1))
-		return 0;
-	if (find_short_packed_object(i, res, sha1))
-		return 0;
-	return -1;
-}
-
-/*
- * This is like "get_sha1()", except it allows "sha1 expressions",
- * notably "xyz^" for "parent of xyz"
- */
-static int get_extended_sha1(char *name, unsigned char *sha1)
-{
-	int parent, ret;
-	int len = strlen(name);
-
-	parent = 1;
-	if (len > 2 && name[len-1] >= '0' && name[len-1] <= '9') {
-		parent = name[len-1] - '0';
-		len--;
-	}
-	if (len > 1 && name[len-1] == '^') {
-		name[len-1] = 0;
-		ret = get_parent(name, sha1, parent);
-		name[len-1] = '^';
-		if (!ret)
-			return 0;
-	}
-	ret = get_sha1(name, sha1);
-	if (!ret)
-		return 0;
-	return get_short_sha1(name, sha1);
-}
-
 static void show_default(void)
 {
 	char *s = def;
@@ -291,7 +113,7 @@ static void show_default(void)
 		unsigned char sha1[20];
 
 		def = NULL;
-		if (!get_extended_sha1(s, sha1)) {
+		if (!get_sha1(s, sha1)) {
 			show_rev(NORMAL, sha1);
 			return;
 		}
@@ -372,10 +194,10 @@ int main(int argc, char **argv)
 			unsigned char end[20];
 			char *n = dotdot+2;
 			*dotdot = 0;
-			if (!get_extended_sha1(arg, sha1)) {
+			if (!get_sha1(arg, sha1)) {
 				if (!*n)
 					n = "HEAD";
-				if (!get_extended_sha1(n, end)) {
+				if (!get_sha1(n, end)) {
 					if (no_revs)
 						continue;
 					def = NULL;
@@ -386,14 +208,14 @@ int main(int argc, char **argv)
 			}
 			*dotdot = '.';
 		}
-		if (!get_extended_sha1(arg, sha1)) {
+		if (!get_sha1(arg, sha1)) {
 			if (no_revs)
 				continue;
 			def = NULL;
 			show_rev(NORMAL, sha1);
 			continue;
 		}
-		if (*arg == '^' && !get_extended_sha1(arg+1, sha1)) {
+		if (*arg == '^' && !get_sha1(arg+1, sha1)) {
 			if (no_revs)
 				continue;
 			def = NULL;
diff --git a/sha1_file.c b/sha1_file.c
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -46,21 +46,6 @@ int get_sha1_hex(const char *hex, unsign
 	return 0;
 }
 
-static int get_sha1_file(const char *path, unsigned char *result)
-{
-	char buffer[60];
-	int fd = open(path, O_RDONLY);
-	int len;
-
-	if (fd < 0)
-		return -1;
-	len = read(fd, buffer, sizeof(buffer));
-	close(fd);
-	if (len < 40)
-		return -1;
-	return get_sha1_hex(buffer, result);
-}
-
 static char *git_dir, *git_object_dir, *git_index_file, *git_refs_dir,
 	*git_graft_file;
 static void setup_git_env(void)
@@ -132,30 +117,6 @@ int safe_create_leading_directories(char
 	return 0;
 }
 
-int get_sha1(const char *str, unsigned char *sha1)
-{
-	static const char *prefix[] = {
-		"",
-		"refs",
-		"refs/tags",
-		"refs/heads",
-		"refs/snap",
-		NULL
-	};
-	const char **p;
-
-	if (!get_sha1_hex(str, sha1))
-		return 0;
-
-	for (p = prefix; *p; p++) {
-		char * pathname = git_path("%s/%s", *p, str);
-		if (!get_sha1_file(pathname, sha1))
-			return 0;
-	}
-
-	return -1;
-}
-
 char * sha1_to_hex(const unsigned char *sha1)
 {
 	static char buffer[50];
diff --git a/sha1_name.c b/sha1_name.c
new file mode 100644
--- /dev/null
+++ b/sha1_name.c
@@ -0,0 +1,222 @@
+#include "cache.h"
+#include "commit.h"
+
+static int find_short_object_filename(int len, const char *name, unsigned char *sha1)
+{
+	static char dirname[PATH_MAX];
+	char hex[40];
+	DIR *dir;
+	int found;
+
+	snprintf(dirname, sizeof(dirname), "%s/%.2s", get_object_directory(), name);
+	dir = opendir(dirname);
+	sprintf(hex, "%.2s", name);
+	found = 0;
+	if (dir) {
+		struct dirent *de;
+		while ((de = readdir(dir)) != NULL) {
+			if (strlen(de->d_name) != 38)
+				continue;
+			if (memcmp(de->d_name, name + 2, len-2))
+				continue;
+			memcpy(hex + 2, de->d_name, 38);
+			if (++found > 1)
+				break;
+		}
+		closedir(dir);
+	}
+	if (found == 1)
+		return get_sha1_hex(hex, sha1) == 0;
+	return 0;
+}
+
+static int match_sha(unsigned len, const unsigned char *a, const unsigned char *b)
+{
+	do {
+		if (*a != *b)
+			return 0;
+		a++;
+		b++;
+		len -= 2;
+	} while (len > 1);
+	if (len)
+		if ((*a ^ *b) & 0xf0)
+			return 0;
+	return 1;
+}
+
+static int find_short_packed_object(int len, const unsigned char *match, unsigned char *sha1)
+{
+	struct packed_git *p;
+
+	prepare_packed_git();
+	for (p = packed_git; p; p = p->next) {
+		unsigned num = num_packed_objects(p);
+		unsigned first = 0, last = num;
+		while (first < last) {
+			unsigned mid = (first + last) / 2;
+			unsigned char now[20];
+			int cmp;
+
+			nth_packed_object_sha1(p, mid, now);
+			cmp = memcmp(match, now, 20);
+			if (!cmp) {
+				first = mid;
+				break;
+			}
+			if (cmp > 0) {
+				first = mid+1;
+				continue;
+			}
+			last = mid;
+		}
+		if (first < num) {
+			unsigned char now[20], next[20];
+			nth_packed_object_sha1(p, first, now);
+			if (match_sha(len, match, now)) {
+				if (nth_packed_object_sha1(p, first+1, next) || !match_sha(len, match, next)) {
+					memcpy(sha1, now, 20);
+					return 1;
+				}
+			}
+		}
+	}
+	return 0;
+}
+
+static int get_short_sha1(const char *name, unsigned char *sha1)
+{
+	int i;
+	char canonical[40];
+	unsigned char res[20];
+
+	memset(res, 0, 20);
+	memset(canonical, 'x', 40);
+	for (i = 0;;i++) {
+		unsigned char c = name[i];
+		unsigned char val;
+		if (!c || i > 40)
+			break;
+		if (c >= '0' && c <= '9')
+			val = c - '0';
+		else if (c >= 'a' && c <= 'f')
+			val = c - 'a' + 10;
+		else if (c >= 'A' && c <='F') {
+			val = c - 'A' + 10;
+			c -= 'A' - 'a';
+		}
+		else
+			return -1;
+		canonical[i] = c;
+		if (!(i & 1))
+			val <<= 4;
+		res[i >> 1] |= val;
+	}
+	if (i < 4)
+		return -1;
+	if (find_short_object_filename(i, canonical, sha1))
+		return 0;
+	if (find_short_packed_object(i, res, sha1))
+		return 0;
+	return -1;
+}
+
+static int get_sha1_file(const char *path, unsigned char *result)
+{
+	char buffer[60];
+	int fd = open(path, O_RDONLY);
+	int len;
+
+	if (fd < 0)
+		return -1;
+	len = read(fd, buffer, sizeof(buffer));
+	close(fd);
+	if (len < 40)
+		return -1;
+	return get_sha1_hex(buffer, result);
+}
+
+static int get_sha1_basic(const char *str, int len, unsigned char *sha1)
+{
+	static const char *prefix[] = {
+		"",
+		"refs",
+		"refs/tags",
+		"refs/heads",
+		"refs/snap",
+		NULL
+	};
+	const char **p;
+
+	if (!get_sha1_hex(str, sha1))
+		return 0;
+
+	for (p = prefix; *p; p++) {
+		char *pathname = git_path("%s/%.*s", *p, len, str);
+		if (!get_sha1_file(pathname, sha1))
+			return 0;
+	}
+
+	return -1;
+}
+
+static int get_sha1_1(const char *name, int len, unsigned char *sha1);
+
+static int get_parent(const char *name, int len,
+		      unsigned char *result, int idx)
+{
+	unsigned char sha1[20];
+	int ret = get_sha1_1(name, len, sha1);
+	struct commit *commit;
+	struct commit_list *p;
+
+	if (ret)
+		return ret;
+	commit = lookup_commit_reference(sha1);
+	if (!commit)
+		return -1;
+	if (parse_commit(commit))
+		return -1;
+	if (!idx) {
+		memcpy(result, commit->object.sha1, 20);
+		return 0;
+	}
+	p = commit->parents;
+	while (p) {
+		if (!--idx) {
+			memcpy(result, p->item->object.sha1, 20);
+			return 0;
+		}
+		p = p->next;
+	}
+	return -1;
+}
+
+static int get_sha1_1(const char *name, int len, unsigned char *sha1)
+{
+	int parent, ret;
+
+	parent = 1;
+	if (len > 2 && name[len-1] >= '0' && name[len-1] <= '9') {
+		parent = name[len-1] - '0';
+		len--;
+	}
+	if (len > 1 && name[len-1] == '^') {
+		ret = get_parent(name, len-1, sha1, parent);
+		if (!ret)
+			return 0;
+	}
+	ret = get_sha1_basic(name, len, sha1);
+	if (!ret)
+		return 0;
+	return get_short_sha1(name, sha1);
+}
+
+/*
+ * This is like "get_sha1_basic()", except it allows "sha1 expressions",
+ * notably "xyz^" for "parent of xyz"
+ */
+int get_sha1(const char *name, unsigned char *sha1)
+{
+	return get_sha1_1(name, strlen(name), sha1);
+}

Re: [PATCH] (preview) Renaming push.

[Linus Torvalds]

On Wed, 3 Aug 2005, Junio C Hamano wrote:
> 
> While I have not updated the send-pack <src>:<dst> syntax, I
> added a horrible hack that some people may love to see.  This
> removes the need to use git-rev-parse from many commands.

Yes, I think this makes sense. We had three different sha1 parsers: 
get_sha1(), get_sha1_hex(), and get_extended_sha1().

None of the users of get_sha1() really have any reason to want the limited
form, so I think your patch does the right thing.

Now, for extra bonus points, maybe you should make "git-rev-list" also 
understand the "rev..rev" format (which you can't do with just the 
get_sha1() interface, since it expands into more).

Of course, most people don't tend to use git-rev-list directly, so maybe 
that's not a biggie, and so git-rev-parse is fine.

		Linus

Re: [PATCH] (preview) Renaming push.

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> Now, for extra bonus points, maybe you should make "git-rev-list" also 
> understand the "rev..rev" format (which you can't do with just the 
> get_sha1() interface, since it expands into more).

Hmph.  That makes sense.

What I set out to do when I started ripping extended_sha1 out of
rev-parse was actually something else.

Remember I was stupidly wondering why "git log master..gitk" did
not do what I wanted to?

The reason I started looking at git-rev-parse was because I
wanted to add "git log master...gitk" (three dots) which is
equivalent to "git log $(git-merge-base master gitk)..gitk".
That was what I wanted to have when I asked that stupid
question.

But unfortunately my GIT day this week is over, so the rest is
for the weekend.

[PATCH] Teach rev-list since..til notation.

[Junio C Hamano]

The King Penguin says:

    Now, for extra bonus points, maybe you should make "git-rev-list" also
    understand the "rev..rev" format (which you can't do with just the
    get_sha1() interface, since it expands into more).

The faithful servant makes it so.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 rev-list.c |   38 ++++++++++++++++++++++++++++++--------
 1 files changed, 30 insertions(+), 8 deletions(-)

f22d271a469b65760f73dc483aeb2448619e0ff1
diff --git a/rev-list.c b/rev-list.c
--- a/rev-list.c
+++ b/rev-list.c
@@ -457,6 +457,15 @@ static struct commit *get_commit_referen
 	die("%s is unknown object", name);
 }
 
+static void handle_one_commit(struct commit *com, struct commit_list **lst)
+{
+	if (!com || com->object.flags & SEEN)
+		return;
+	com->object.flags |= SEEN;
+	commit_list_insert(com, lst);
+}
+
+
 int main(int argc, char **argv)
 {
 	struct commit_list *list = NULL;
@@ -465,6 +474,7 @@ int main(int argc, char **argv)
 	for (i = 1 ; i < argc; i++) {
 		int flags;
 		char *arg = argv[i];
+		char *dotdot;
 		struct commit *commit;
 
 		if (!strncmp(arg, "--max-count=", 12)) {
@@ -523,21 +533,33 @@ int main(int argc, char **argv)
 			continue;
 		}
 
+		if (show_breaks && !merge_order)
+			usage(rev_list_usage);
+
 		flags = 0;
+		dotdot = strstr(arg, "..");
+		if (dotdot) {
+			char *next = dotdot + 2;
+			struct commit *exclude = NULL;
+			struct commit *include = NULL;
+			*dotdot = 0;
+			exclude = get_commit_reference(arg, UNINTERESTING);
+			include = get_commit_reference(next, 0);
+			if (exclude && include) {
+				limited = 1;
+				handle_one_commit(exclude, &list);
+				handle_one_commit(include, &list);
+				continue;
+			}
+			*next = '.';
+		}
 		if (*arg == '^') {
 			flags = UNINTERESTING;
 			arg++;
 			limited = 1;
 		}
-		if (show_breaks && !merge_order)
-			usage(rev_list_usage);
 		commit = get_commit_reference(arg, flags);
-		if (!commit)
-			continue;
-		if (commit->object.flags & SEEN)
-			continue;
-		commit->object.flags |= SEEN;
-		commit_list_insert(commit, &list);
+		handle_one_commit(commit, &list);
 	}
 
 	if (!merge_order) {		

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Johannes Schindelin wrote:
> 
> On Wed, 3 Aug 2005, Linus Torvalds wrote:
> 
> > Are you sure you have a good git version on master? I've never seen
> > anything like that, and I push all the time..
> 
> Call him Zaphod: he has two heads (master and pu). You don't.

Oh, but I most definitely do. I update several heads at a time when I 
create a new tag etc.

What I don't do is to rewrite my history, though..

		Linus

Re: Users of git-check-files?

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> Are you sure you have a good git version on master? I've never seen 
> anything like that, and I push all the time..

I have been esuspecting that it happens only because I rewind
and rebase "pu", which you never do.  The thing is, even though
I rewind "pu" all the time, it happens only occasionally.

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Junio C Hamano wrote:
>
> Linus Torvalds <torvalds@osdl.org> writes:
> 
> > Are you sure you have a good git version on master? I've never seen 
> > anything like that, and I push all the time..
> 
> I have been esuspecting that it happens only because I rewind
> and rebase "pu", which you never do.  The thing is, even though
> I rewind "pu" all the time, it happens only occasionally.

Oh, that would do it. 

You need to prune back the remote tree you send to, so that it is a real
subset of what you are sending from (at least as far as the branch you
send is concerned - other branches may be ahead of you, of course). Unlike
"git-fetch-pack", the send-pack interface does not do any common commit
discovery, so if you have state on the other end that isn't on your local
end, that's setting yourself up for problems: you are basically misusing
the interfaces.

I started out to make the "-f" flag to send-file work around it, but I
never finished that, partly because it really ends up being the same thing
as "git-fetch-pack" in reverse, which was against the whole point of
git-send-pack. Send-pack is meant to be an "update remote tree" thing, 
with the assumption that the remote tree is a subset - and exactly that 
assumption is what makes send-pack much cheaper than fetch-pack.

(Actually, to me it's not the "cheaper" that matters, but exactly the fact
that send-pack is so safe - if somebody has changed something at the other 
end that I don't have in mine, I _want_ errors, because that would be a 
serious problem).

		Linus

Re: Users of git-check-files?

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> I started out to make the "-f" flag to send-file work around it, but I
> never finished that, partly because it really ends up being the same thing
> as "git-fetch-pack" in reverse, which was against the whole point of
> git-send-pack. Send-pack is meant to be an "update remote tree" thing, 
> with the assumption that the remote tree is a subset - and exactly that 
> assumption is what makes send-pack much cheaper than fetch-pack.

I think in addition to the existing ref_newer() check, which
makes sure you are advancing the remote head, not just replacing
with something unrelated, making sure that we have objects
referenced by ref->old_sha1 we obtained from the remote on our
end for all branches involved would be the only thing needed.
The latter the users should not even be able to override with
the --force flag, of course, but we would remind them to pull
from the other end first.

Re: Users of git-check-files?

[Josef Weidendorfer]

On Wednesday 03 August 2005 19:37, you wrote:
> Linus Torvalds <torvalds@osdl.org> writes:
> > I started out to make the "-f" flag to send-file work around it, but I
> > never finished that, partly because it really ends up being the same
> > thing as "git-fetch-pack" in reverse, which was against the whole point
> > of git-send-pack. Send-pack is meant to be an "update remote tree" thing,
> > with the assumption that the remote tree is a subset - and exactly that
> > assumption is what makes send-pack much cheaper than fetch-pack.
>
> I think in addition to the existing ref_newer() check, which
> makes sure you are advancing the remote head, not just replacing
> with something unrelated, making sure that we have objects
> referenced by ref->old_sha1 we obtained from the remote on our
> end for all branches involved would be the only thing needed.
> The latter the users should not even be able to override with
> the --force flag, of course, but we would remind them to pull
> from the other end first.

But my example shows that the error happens even with 2 branches totally 
unrelated to each other: if branch1 got a new commit, you can not push to
branch2 from another clone.

Why is it not enough to have all the history of a remote branch in the local 
clone to be able to push to this branch?

Josef

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Josef Weidendorfer wrote:
> 
> But my example shows that the error happens even with 2 branches totally 
> unrelated to each other: if branch1 got a new commit, you can not push to
> branch2 from another clone.

Sure you can.

	git-send-pack remote branch2

and you've just done so.

The shorthand of "no branches listed" expands to _all_ branches, and if 
you try to send all branches, then you're trying to update "branch1" with 
something you don't have. That's your usage error.

		Linus

Re: Users of git-check-files?

[Linus Torvalds]

On Wed, 3 Aug 2005, Junio C Hamano wrote:
> 
> I think in addition to the existing ref_newer() check, which
> makes sure you are advancing the remote head, not just replacing
> with something unrelated, making sure that we have objects
> referenced by ref->old_sha1 we obtained from the remote on our
> end for all branches involved would be the only thing needed.

Yes.

		Linus

Re: Users of git-check-files?

[Junio C Hamano]

This patch seems to fix the problem.

 * If the original value of remote ref refers to an object we do
   not have, and if the ref is one of the branches we are trying
   to push, we refuse to update it.

 * Otherwise, we do not attempt to use such an value when
   computing what objects to put in pack, since rev-list would
   fail.

I've tested Josef's two-branches case, one repo updates one
branch and pushes to the central repo, another repo updates the
other branch and pushed to the central repo.  The old code
barfed when invoking rev-list, but this does not seem to.

Josef, could you give it a try please?

---
cd /opt/packrat/playpen/public/in-place/git/git.junio/
jit-diff
# - master: git-send-email-script: minimum whitespace cleanup.
# + (working tree)
diff --git a/send-pack.c b/send-pack.c
--- a/send-pack.c
+++ b/send-pack.c
@@ -43,7 +43,8 @@ static void exec_rev_list(struct ref *re
 		char *buf = malloc(100);
 		if (i > 900)
 			die("git-rev-list environment overflow");
-		if (!is_zero_sha1(refs->old_sha1)) {
+		if (!is_zero_sha1(refs->old_sha1) &&
+		    has_sha1_file(refs->old_sha1)) {
 			args[i++] = buf;
 			snprintf(buf, 50, "^%s", sha1_to_hex(refs->old_sha1));
 			buf += 50;
@@ -208,6 +209,12 @@ static int send_pack(int in, int out, in
 			continue;
 		}
 
+		if (!has_sha1_file(ref->old_sha1)) {
+			error("remote '%s' object %s does not exist on local",
+			      name, sha1_to_hex(ref->old_sha1));
+			continue;
+		}
+
 		/* Ok, mark it for update */
 		memcpy(ref->new_sha1, new_sha1, 20);
 	}

Compilation finished at Wed Aug  3 11:02:15

Re: Users of git-check-files?

[Josef Weidendorfer]

On Wednesday 03 August 2005 20:07, Junio C Hamano wrote:
> Josef, could you give it a try please?

Perfect. Thanks.

Josef

Re: Users of git-check-files?

[Johannes Schindelin]

Hi,

On Wed, 3 Aug 2005, Junio C Hamano wrote:

> Sorry, you have not lost anything.  It is my bad that this is
> the first time I brought it up.  I've been seeing that from time
> to time when I push to either my "send to master" repository
> from my working repository, or from the "send to master"
> repository to master.kernel.org, but I haven't figured it out if
> there is any pattern.

This could be related to what I was realizing the other day: when trying 
to push to a repository (just one branch), but I do not have _all_ of the 
remote branches pulled, then it fails.

Ciao,
Dscho

Re: Users of git-check-files?

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> On Tue, 2 Aug 2005, Junio C Hamano wrote:
>> 
>> How about git-rev-tree?  Does anybody care?
>
> Yeah, probably not. git-rev-list does so much more than git-rev-tree ever 
> did.

I will keep git-rev-list; used in Jeff's git-changes-script and
some parts of Cogito as well.

Re: Users of git-check-files?

[Johannes Schindelin]

Hi,

On Thu, 4 Aug 2005, Junio C Hamano wrote:

> I will keep git-rev-list; used in Jeff's git-changes-script and
> some parts of Cogito as well.

According to my grep's, these files use git-rev-list:

git-bisect-script
git-cherry
git-format-patch-script
git-log-script
git-repack-script
git-whatchanged
gitk
send-pack.c
upload-pack.c

So better keep it :-)

Ciao,
Dscho