[RFC] faking cvs annotate

[RFC] faking cvs annotate

[Martin Langhoff]

I know, I know, use pickaxe :-)

Any suggestions as to how to implement a not-too-slow annotate that
looks reasonable enough so as to fool real cvs clients? Let's assume
I'll have the "per-file-version numbers" to translate commit SHA1s
into cvs-ish version numbers in a magic hat, right next to the fluffy
bunnies.

Suggestions of GIT machinery that would shortcut the trip from

     git-rev-list HEAD $path

to a annotate-ish output. Did I dream it or is qgit showing something
annotate-ish in its screenshots?

Note! Just told a white lie: I won't be actually using git-rev-list.
Instead, I'll have a rev list that will gloss over branched
development, picking only one "path" of the history every time
(initially using the algorithm formerly known as rand() ) , and
showing merge commits as one.

cheers,


martin

Re: [RFC] faking cvs annotate

[Johannes Schindelin]

Hi,

On Fri, 16 Dec 2005, Martin Langhoff wrote:

> I know, I know, use pickaxe :-)
> 
> Any suggestions as to how to implement a not-too-slow annotate that
> looks reasonable enough so as to fool real cvs clients?

For starters, you could use my attempt:

http://www.gelato.unsw.edu.au/archives/git/0508/7171.html

It does not fool any CVS user, because it does not fake version numbers. 
Instead, you can display SHA1s, short SHA1s, author names, or mixtures 
thereof.

However, I learnt to use git-whatchanged in the meantime, and I'll 
probably never go back.

Hth,
Dscho

Re: [RFC] faking cvs annotate

[Martin Langhoff]

On 12/16/05, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> For starters, you could use my attempt:
> http://www.gelato.unsw.edu.au/archives/git/0508/7171.html

Not bad at all! I might simplify it a bit (won't need the commitmsg
parsing), and see if how fast I can make it.

> It does not fool any CVS user,

Ah, no. Those have been fooled already ;-)

> However, I learnt to use git-whatchanged in the meantime, and I'll
> probably never go back.

Of course. I'm trying to meet a few the expectations of popular cvs
clients (cli, TortoiseCVS, Eclipse, etc). If a workable git-annotate
falls out of this, oh well.

OTOH, yours didn't get included. Any particular reasons? (other than
stressing the point that you should use whatchanged/pickaxe?)

Also! On 12/16/05, Junio C Hamano <junkio@cox.net> wrote:
> I haven't actively done anything but one of the good things that
> could happen is to split out the access routines for annotate
> database qgit build when run the first time in the repository,
> and make them available to other Porcelains.  There is no need
> to reinvent the wheel.

That would be cool! Though my particular implementation may be a tad
unconventional in its view of the "file history" as a strictly linear
thing. Normal git porcelains don't even blink when dealing with
branching/merging development histories.

cheers,


martin

Re: [RFC] faking cvs annotate

[Linus Torvalds]

On Fri, 16 Dec 2005, Martin Langhoff wrote:
>
> On 12/16/05, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > For starters, you could use my attempt:
> > http://www.gelato.unsw.edu.au/archives/git/0508/7171.html
> 
> Not bad at all! I might simplify it a bit (won't need the commitmsg
> parsing), and see if how fast I can make it.

NOTE! You can make it a whole lot faster these days by passing the path 
down to git-rev-list. But as usual, I'm perl-incompatible, so I can't help 
you with that modification.

> OTOH, yours didn't get included. Any particular reasons? (other than
> stressing the point that you should use whatchanged/pickaxe?)

I'd love to see a git-annotate, but I want it to be efficient enough that 
it's not embarrassing.

For example, "git-whatchanged" is certainly efficient enough, and you 
could just parse that output (possible without using -p and comparing file 
contents on your own against the current HEAD as you progress backwards)

Or, you could now use "git-rev-list <rev> <filename>", which is even more 
efficient, but the downside of that is that it computes the _full_ graph 
before it starts outputting the early ones. 

But, for example:

	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
	real    0m37.993s
	user    0m41.912s
	sys     0m0.400s

	[torvalds@g5 linux-history]$ time git-rev-list HEAD drivers/char/Makefile > /dev/null 
	real    0m6.713s
	user    0m6.656s
	sys     0m0.056s

that's the old Linux history thing, with 60,000+ commit. The path-based 
git-rev-list thing cuts it down to just 103 commits you need to look at 
(so the annotate itself should be really fast, and the cost really is all 
in that "git-rev-list" thing).

Of course, with the current kernel tree, the git-rev-list takes a lot less 
time, and only results in six commits, but that's a tree that only has a 
few months of history, so when looking at performance, you really should 
look at some bigger tree.

NOTE! You can also decide to cut down on time by saying "annotate back 
only to a specific date", ie something like

	git-rev-list $(git-rev-parse --since=2.years.ago HEAD) drivers/char/Makefile

which limits the thing by filename _and_ date (a good way to handle really 
really big projects) where the rev-parse might otherwise be very 
expensive.

And, of course, a quality implementation will notice when a file 
disappears, and use "git-diff-tree -M" on that commit to see if it was 
renamed, and then continuing with the old name...

			Linus

Re: [RFC] faking cvs annotate

[Nicolas Pitre]

On Fri, 16 Dec 2005, Linus Torvalds wrote:

> But, for example:
> 
> 	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
> 	real    0m37.993s
> 	user    0m41.912s
> 	sys     0m0.400s

Hmmm... I'd like to have a system like that too, where reality is 
smaller than the sum of system and user time.  ;-)


Nicolas

Re: [RFC] faking cvs annotate

[Linus Torvalds]

On Fri, 16 Dec 2005, Nicolas Pitre wrote:

> On Fri, 16 Dec 2005, Linus Torvalds wrote:
> 
> > But, for example:
> > 
> > 	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
> > 	real    0m37.993s
> > 	user    0m41.912s
> > 	sys     0m0.400s
> 
> Hmmm... I'd like to have a system like that too, where reality is 
> smaller than the sum of system and user time.  ;-)

It's called "smp".

You can buy then anywhere.

		Linus

Re: [RFC] faking cvs annotate

[Nicolas Pitre]

On Fri, 16 Dec 2005, Linus Torvalds wrote:

> 
> 
> On Fri, 16 Dec 2005, Nicolas Pitre wrote:
> 
> > On Fri, 16 Dec 2005, Linus Torvalds wrote:
> > 
> > > But, for example:
> > > 
> > > 	[torvalds@g5 linux-history]$ time git-whatchanged drivers/char/Makefile > /dev/null 
> > > 	real    0m37.993s
> > > 	user    0m41.912s
> > > 	sys     0m0.400s
> > 
> > Hmmm... I'd like to have a system like that too, where reality is 
> > smaller than the sum of system and user time.  ;-)
> 
> It's called "smp".

Gah!  Maybe I should have tried the same before posting, since I get 
this on my P4 with HT:

	real    0m36.930s
	user    0m41.971s
	sys     0m0.656s

/me hides shamefully


Nicolas

Re: [RFC] faking cvs annotate

[Junio C Hamano]

Martin Langhoff <martin.langhoff@gmail.com> writes:

> Suggestions of GIT machinery that would shortcut the trip from
>
>      git-rev-list HEAD $path
>
> to a annotate-ish output. Did I dream it or is qgit showing something
> annotate-ish in its screenshots?

I haven't actively done anything but one of the good things that
could happen is to split out the access routines for annotate
database qgit build when run the first time in the repository,
and make them available to other Porcelains.  There is no need
to reinvent the wheel.

Re: [RFC] faking cvs annotate

[Eric Wong]

Junio C Hamano <junkio@cox.net> wrote:
> Martin Langhoff <martin.langhoff@gmail.com> writes:
> 
> > Suggestions of GIT machinery that would shortcut the trip from
> >
> >      git-rev-list HEAD $path
> >
> > to a annotate-ish output. Did I dream it or is qgit showing something
> > annotate-ish in its screenshots?
> 
> I haven't actively done anything but one of the good things that
> could happen is to split out the access routines for annotate
> database qgit build when run the first time in the repository,
> and make them available to other Porcelains.  There is no need
> to reinvent the wheel.

I started working on something using Algorithm::Annotate on CPAN during
an ADD excursion a few weeks ago, but mostly forgot about it (thanks
again to ADD :).  A few versions of git later and it still seems to
work as well as I remembered it.

It doesn't handle renames/copies yet, but it does let you customize the
left-hand side output unlike most/(all?) annotate implementations.

I guess it's also quite useful when the output is piped to vim with a
repository browser like the one I brought up in
<20051124093322.GA3899@mail.yhbt.net>

---------- 8< ---------

#!/usr/bin/env perl
# Copyright (c) 2005, Eric Wong <normalperson@yhbt.net>
#
# This file is licensed under the GPL v2, or a later version
# at the discretion of Linus.
#
# Read git-whatchanged output and let all Algorithm::Annotate do all
# the hard work for us.  Thanks to Chia-liang Kao for the awesome 
# Algorithm::Annotate module.
#
# Formatting options of annotation output is available.
# Dates can be formatted through strftime strings using the -D switch,
# and annotation expansion can be done using the -F switch
# on following variables:

=cut
	%commit% %tree% %author_name% %author_email% %author_date%
	%committer_name% %committer_email% %commit_date% %message%
=cut
use warnings;
use strict;

sub usage ($) {
	print '* git-annotate ',
		'[-w <width>] [-D <date-format>] [-F <annotate-format>] ',
		'[--localtime] [--line-prefix <str>] [--line-postfix <str>] ',
		'<file>', "\n";
	exit $_[0];
}

# everybody with Perl 5 should have these, right?
use POSIX qw(strftime);
use Getopt::Long qw(:config gnu_getopt no_ignore_case no_auto_abbrev);

my %_o; # options
usage(1) unless (GetOptions(\%_o, qw(help|H|h width|w=i
		line_prefix|line-prefix:s line_postfix|line-postfix:s
		format|F=s date_format|date-format|D=s localtime|l)));
usage(0) if $_o{help};

# ok, see if we can do more than show a help message:
require Algorithm::Annotate or die
		"Can't find Algorithm::Annotate, get it from CPAN ($!)\n";

my $file = shift or usage(1);
unless (-f $file) {
	print STDERR "File does not exist or is not a file: $file\n";
	exit 1;
}

my $date_format = $_o{date_format} || '%Y-%m-%dT%H-%M-%SZ'; # ISO 8601

my $re_sha1 = qr/[a-f0-9]{40}/; # match 40 char hex strings
my @blobs; # we read from newest to oldest, but $ann needs it reversed

sub t ($) {
	return localtime($_[0]) if $_o{'localtime'};
	return gmtime($_[0]);
}

if (my $pid = open my $child, '-|') {
	my $meta;
	while (<$child>) {
		chomp;
		if (/^diff-tree ($re_sha1) /o) {
			$meta = { commit => $1, message => [] };
		} elsif (/^tree ($re_sha1)$/) {
			$meta->{tree} = $1;
		} elsif (/^author (.*) <(\S+)> (\d+) [\+\-]?\d+$/) {
			$meta->{author_name} = $1;
			$meta->{author_email} = $2;
			$meta->{author} = "$1 <$2>";
			$meta->{author_date} = strftime($date_format, t($3));
		} elsif (/^committer (.*) <(\S+)> (\d+) [\+\-]?\d+$/) {
			$meta->{committer_name} = $1;
			$meta->{committer_email} = $2;
			$meta->{committer} = "$1 <$2>";
			$meta->{commit_date} = strftime($date_format, t($3));
		} elsif (/^\s{4}(.*)$/) {
			push @{$meta->{message}}, $1;
		} elsif (/^:\d{6} \d{6} $re_sha1 ($re_sha1)/o) {
			my $blob = $1;
			if ($blob =~ /^0{40}$/) {
				# nonexistent (hopefully!) blob
				$meta = undef;
				next;
			}
			
			$meta->{message} = join("\n",@{$meta->{message}});
			push @blobs, { meta => $meta, blob => $blob };
			$meta = undef;
		}
	}
} else {
	exec('git-whatchanged','-m','--pretty=raw','--',$file) or die
			"Unable to execute git-whatchanged $file: $? $!";
}

my $msg_format = $_o{format} || '%commit% %commit_date%';
my $max_len = 0;
my $ann = Algorithm::Annotate->new;

foreach my $x (reverse @blobs) {
	my @contents;
	
	# no need to do safe pipe opens since we know $x->{blob} is a hex sum
	@contents = `git-cat-file blob $x->{blob}`;

	# $msg is the left hand side of our final annotation output,
	# format it according to $msg_format
	my $msg = $msg_format;
	my $meta = $x->{meta};

	# our annotation message templating engine:
	$msg =~ s/\%$_\%/$meta->{$_}/g foreach keys %$meta;
	$msg =~ s/\\t/\t/g;
	$msg =~ s/\\n/\n/g;

	$ann->add($msg, \@contents);
	if ($msg !~ /\n/) {
		$max_len = length $msg if length $msg > $max_len;
	} else {
		foreach (split /\n/, $msg) {
			$max_len = length $_ if length $_ > $max_len;
		}
	}
}

# see if we can annotate local changes, too
my $local_change;
my @diff_opts = qw(-z -C --find-copies-harder --name-only);
{ # see if we've changed $file in the working tree
	if (my $pid = open my $child, '-|') {
		$local_change = (<$child>);
	} else {
		exec('git-diff-files',@diff_opts,'--',$file);
	}
}

unless ($local_change) {
	# see if we've changed $file in the working tree
	if (my $pid = open my $child, '-|') {
		$local_change = (<$child>);
	} else {
		exec('git-diff-index',@diff_opts,'--cached','HEAD','--',$file);
	}
}

open my $fd,'<',$file or die "Error opening file: $file: $!\n";
my @contents = <$fd>;

if ($local_change) {
	my $msg = '(uncommitted): ';
	$ann->add($msg, \@contents);
	$max_len = length $msg if length $msg > $max_len;
}

# $result here is an array ref, each element corresponding to a line of
# the annotated file contents
my $result = $ann->result or do {
	print STDERR "Unable to annotate $file.  Is it tracked by git?\n";
	exit 1;
};

my $pre = defined $_o{line_prefix} ? $_o{line_prefix} : '';
my $post = defined $_o{line_postfix} ? $_o{line_postfix} : ': ';
my $width = $_o{width} || $max_len;

foreach (@contents) {
	my $msg = shift @$result;
	if ($msg !~ /\n/) {
		print ($pre, pack("A$width", $msg), $post, $_);
	} else {
		my @msgs = split /\n/, $msg;
		print ($pre, pack("A$width", shift @msgs), $post, $_);
		foreach my $m (@msgs) {
			print ($pre, pack("A$width", $m), $post,"\n");
		}
	}
}

close $fd;

exit 0;

-- 
Eric Wong

Re: [RFC] faking cvs annotate

[Ryan Anderson]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

Eric Wong wrote:

> #!/usr/bin/env perl
> # Copyright (c) 2005, Eric Wong <normalperson@yhbt.net>
> #
> # This file is licensed under the GPL v2, or a later version
> # at the discretion of Linus.

Please use full names for this kind of authority delegation.

> # Read git-whatchanged output and let all Algorithm::Annotate do all
> # the hard work for us.  Thanks to Chia-liang Kao for the awesome 
> # Algorithm::Annotate module.
> #

You'll want to add libalgorithm-annotate-perl to the Debian/control
Depends: line when you submit this as a patch.  (I dont' have an
RPM-based machine handy, so I can't doublecheck that.)

I was going to comment on the rest, but I don't really have time to play
with it at the moment.


-- 

Ryan Anderson
  sometimes Pug Majere

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]