* Re: Following renames
From: Petr Baudis @ 2006-03-26 22:31 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ryan Anderson, git
In-Reply-To: <Pine.LNX.4.64.0603261415390.15714@g5.osdl.org>
Dear diary, on Mon, Mar 27, 2006 at 12:22:04AM CEST, I got a letter
where Linus Torvalds <torvalds@osdl.org> said that...
> So commit "6" is uninteresting, and commit "5" will never even be
> looked at, since we decided that the history of "d" comes from the
> first parent with the same contents.
And this is the thing I have a problem with - this does not make much
sense to me, why can't we just follow all parents instead of arbitrarily
choosing one of them?
> which is correct (now, there are other histories _too_ that get us to the
> same point, but the one you found this way was _a_ history).
Ok, in that case I want the _full_ history. :-)
> No, it's the expected output just because you expected merges to always
> show up. Merges get ignored if any of the parents have the same content
> already.
Eek. Can I avoid that? What was the reason for choosing this behavior?
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time. I think
I have forgotten this before.
^ permalink raw reply
* Re: Following renames
From: Junio C Hamano @ 2006-03-26 22:43 UTC (permalink / raw)
To: Petr Baudis; +Cc: git, Linus Torvalds
In-Reply-To: <20060326223154.GU18185@pasky.or.cz>
Petr Baudis <pasky@suse.cz> writes:
>> No, it's the expected output just because you expected merges to always
>> show up. Merges get ignored if any of the parents have the same content
>> already.
>
> Eek. Can I avoid that? What was the reason for choosing this behavior?
Perhaps rev-list --sparse?
^ permalink raw reply
* Re: Following renames
From: Linus Torvalds @ 2006-03-26 23:09 UTC (permalink / raw)
To: Petr Baudis; +Cc: Ryan Anderson, git
In-Reply-To: <20060326223154.GU18185@pasky.or.cz>
On Mon, 27 Mar 2006, Petr Baudis wrote:
> Dear diary, on Mon, Mar 27, 2006 at 12:22:04AM CEST, I got a letter
> where Linus Torvalds <torvalds@osdl.org> said that...
> > So commit "6" is uninteresting, and commit "5" will never even be
> > looked at, since we decided that the history of "d" comes from the
> > first parent with the same contents.
>
> And this is the thing I have a problem with - this does not make much
> sense to me, why can't we just follow all parents instead of arbitrarily
> choosing one of them?
Sure, you can. It's _usually_ a huge waste of time, though. Why would you
want to do more work than you need, since clearly the other parent was
_not_ interesting from the standpoint of the question "where did this
content come from"?
> > No, it's the expected output just because you expected merges to always
> > show up. Merges get ignored if any of the parents have the same content
> > already.
>
> Eek. Can I avoid that? What was the reason for choosing this behavior?
Huge efficiency gains.
Lookie here. Do
gitk -- rev-list.c
on the git archive with the current git-rev-list, and with your hacked-up
version.
And tell me my version isn't a hell of a lot better. Because, I guarantee
you, it is. We're just not _interested_ in all those merges that didn't
actually make any difference.
Read up on what modern neuro-science thinks about the human brain, and
what a lot of it is about. It's about ignoring irrelevant information.
The ability to throw stuff out that isn't interesting is the _real_ basis
of true intelligence. I'd rather have git do the _intelligent_ history,
than show history that isn't relevant and workign harder doing so.
Linus
^ permalink raw reply
* Re: Following renames
From: Linus Torvalds @ 2006-03-26 23:10 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Petr Baudis, git
In-Reply-To: <7vodzsq12g.fsf@assigned-by-dhcp.cox.net>
On Sun, 26 Mar 2006, Junio C Hamano wrote:
> Petr Baudis <pasky@suse.cz> writes:
>
> >> No, it's the expected output just because you expected merges to always
> >> show up. Merges get ignored if any of the parents have the same content
> >> already.
> >
> > Eek. Can I avoid that? What was the reason for choosing this behavior?
>
> Perhaps rev-list --sparse?
No. "--sparse" still removes the uninteresting parents of merges. It just
doesn't then make the linear history any denser.
Linus
^ permalink raw reply
* [PATCH] Remove dependency on a file named "-lz"
From: Johannes Schindelin @ 2006-03-26 23:14 UTC (permalink / raw)
To: git, junkio
By changing the dependency "$(LIB_H)" to "$(LIBS)", at least one version
of make thought that a file named "-lz" would be needed.
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
---
Makefile | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
5a8333baa1845924348b958208bee59831d4e04e
diff --git a/Makefile b/Makefile
index a8cb0af..055d155 100644
--- a/Makefile
+++ b/Makefile
@@ -214,8 +214,8 @@
fetch-clone.o revision.o pager.o \
$(DIFF_OBJS)
-LIBS = $(LIB_FILE) $(XDIFF_LIB)
-LIBS += -lz
+GITLIBS = $(LIB_FILE) $(XDIFF_LIB)
+LIBS = $(GITLIBS) -lz
#
# Platform specific tweaks
@@ -554,7 +554,7 @@
-DDEFAULT_GIT_TEMPLATE_DIR='"$(template_dir_SQ)"' $*.c
$(LIB_OBJS): $(LIB_H)
-$(patsubst git-%$X,%.o,$(PROGRAMS)): $(LIBS)
+$(patsubst git-%$X,%.o,$(PROGRAMS)): $(GITLIBS)
$(DIFF_OBJS): diffcore.h
$(LIB_FILE): $(LIB_OBJS)
--
1.2.0.gd95e-dirty
^ permalink raw reply related
* Re: Following renames
From: Petr Baudis @ 2006-03-26 23:26 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Ryan Anderson, git
In-Reply-To: <20060326191445.GQ18185@pasky.or.cz>
Dear diary, on Sun, Mar 26, 2006 at 09:14:45PM CEST, I got a letter
where Petr Baudis <pasky@suse.cz> said that...
> Dear diary, on Sun, Mar 26, 2006 at 06:33:13PM CEST, I got a letter
> where Linus Torvalds <torvalds@osdl.org> said that...
> > If you do
> >
> > git-rev-list --parents --remove-empty $REV -- $filename
> >
> > then you'll get the whole history for that filename. When it ends, you
> > know the file went away, and then you do basically _one_ "where the hell
> > did it go" thing.
> >
> > And yes, it's not git-ls-tree (unless you only want to follow pure
> > renames), it's actually one "git-diff-tree -M $lastrev". Then you just
> > continue with the new filename (and do another "git-rev-list" until you
> > hit the next rename).
>
> I wrote a long rant but then it all suddenly fit together and I have now
> an idea how to implement it reasonably elegantly.
So, this is what I have. Testing (I've gave it very little of that) and
thoughts welcome. It is probably pretty efficient, at least in terms of
fork()s it does only 2*N of them where N is the number of commits
containing interesting renames. Actually, this should be even possible
to reduce to N+1 if you do a single git-diff-tree call and multiplex
different git-rev-lists to it, but I'm too tired to do the trickery now.
It has 'cg' in the name but depends on no Cogito stuff; it should be in
fact possible to trivially put it to git-whatchanged in place of the
final pipeline (not that I'd be suggesting this to be done universally,
but perhaps git-whatchanged -f ...?). There are three downsides in this
regard:
(i) No -c support. I need the separate deltas coming out from
git-diff-tree but I think I can join them together pretty easily on my
own, except that I have problems with -c (see
<20060326102100.GF18185@pasky.or.cz>) so I'm not sure how exactly is it
supposed to behave.
(ii) Only --pretty=raw output. It shouldn't be hard to add the
reformatting code, but I'm personally not going to use it and kind of
lazy, so I'll let someone else do that, I guess. :-)
(iii) Raw deltas required. -p parsing support would be certainly useful
and possible, but see (ii).
To quickly see what it does, you can try it e.g. on the git-log.sh file
in the Git repository.
Thoughts? Opinions? Bugs? Patches?
Signed-off-by: Petr Baudis <pasky@suse.cz>
diff --git a/cg-Xfollowrenames b/cg-Xfollowrenames
new file mode 100755
index 0000000..fa5c552
--- /dev/null
+++ b/cg-Xfollowrenames
@@ -0,0 +1,246 @@
+#!/usr/bin/env perl
+#
+# git-rev-list | git-diff-tree --stdin following renames
+# Copyright (c) Petr Baudis, 2006
+# Uses bits of git-annotate.perl by Ryan Anderson.
+#
+# This script will efficiently show output as of the
+#
+# git-rev-list --remove-empty ARGS -- FILE... |
+# git-diff-tree -M -r -m --stdin --pretty=raw ARGS
+#
+# pipeline, except that it follows renames of individual files listed
+# in the FILE... set.
+#
+# Usage:
+#
+# cg-Xfollowrenames revlistargs -- difftreeargs -- revs -- files
+
+# TODO: Does not work on multiple files properly yet - most probably
+# (I didn't test it!). We want git-rev-list to stop traversing the history
+# when _any_ file disappears while now it probably stops traversing when
+# _all_ files disappear.
+
+use warnings;
+use strict;
+
+$| = 1;
+
+our (@revlist_args, @difftree_args, @revs, @files);
+
+{ # Load arguments
+ my @argp = (\@revlist_args, \@difftree_args, \@revs, \@files);
+ my $argi = 0;
+ for my $arg (@ARGV) {
+ if ($arg eq '--' and $argi < $#argp) {
+ $argi++;
+ next;
+ }
+ push(@{$argp[$argi]}, $arg);
+ }
+}
+
+
+# The heads we watch (sorted by commit time)
+our @heads;
+# Each head is: {
+# # Persistent for the whole line of development:
+# pipe => $pipe,
+# files => \@files, # to watch for
+#
+# id => $sha1, # useful actually only for debugging
+# time => $timestamp,
+# str => $prettyoutput,
+# parents => \@sha1s,
+#
+# # When the commit is processed, spawn these extra heads:
+# recurse => {$sha1id => \@files, ...},
+# }
+
+# To avoid printing duplicate commits
+# FIXME: Currently, we will not handle merge commits properly since
+# we hit them multiple times.
+our %commits;
+
+
+sub open_pipe($@) {
+ my ($stdin, @execlist) = @_;
+
+ my $pid = open my $kid, "-|";
+ defined $pid or die "Cannot fork: $!";
+
+ unless ($pid) {
+ if (defined $stdin) {
+ open STDIN, "<&", $stdin or die "Cannot dup(): $!";
+ }
+ exec @execlist;
+ die "Cannot exec @execlist: $!";
+ }
+
+ return $kid;
+}
+
+sub revlist($@) {
+ my ($rev, @files) = @_;
+ open_pipe(undef, "git-rev-list", "--remove-empty",
+ @revlist_args, $rev, "--", @files)
+ or die "Failed to exec git-rev-list: $!";
+}
+
+sub difftree($) {
+ my ($revlist) = @_;
+ open_pipe($revlist, "git-diff-tree", "-r", "-m", "--stdin", "-M",
+ "--pretty=raw", @difftree_args)
+ or die "Failed to exec git-diff-tree: $!";
+}
+
+sub revdiffpipe($@) {
+ my ($rev, @files) = @_;
+ my $pipe = difftree(revlist($rev, @files));
+}
+
+
+sub read_commit($$) {
+ my ($head, $tolerant) = @_;
+ my $pipe = $head->{'pipe'};
+ my $against;
+ my @oldset = @{$head->{'files'}};
+ my @newset;
+ my $rename;
+
+ # Load header
+ while (my $line = <$pipe>) {
+ $head->{'str'} .= $line;
+ chomp $line;
+ $line eq '' and goto header_loaded;
+
+ if ($line =~ /^diff-tree (\S+) \(from (root|\S+)\)/) {
+ $head->{'id'} = $1;
+ if (not $tolerant and $commits{$1}++) {
+ close $pipe;
+ return undef;
+ }
+ # The 'root' case is harmless since there'll be no renames.
+ $against = $2;
+ } elsif ($line =~ /^parent (\S+)/) {
+ push (@{$head->{'parents'}}, $1);
+ } elsif ($line =~ /^committer .*?> (\d+)/) {
+ $head->{'time'} = $1;
+ }
+ }
+ return undef;
+header_loaded:
+
+ # Load message
+ while (my $line = <$pipe>) {
+ $head->{'str'} .= $line;
+ chomp $line;
+ $line eq '' and goto message_loaded;
+ }
+ return undef;
+message_loaded:
+
+ # Load delta
+ while (my $line = <$pipe>) {
+ $head->{'str'} .= $line;
+ chomp $line;
+ $line eq '' and goto delta_loaded;
+
+ $line =~ /^:/ or return undef;
+ my ($info, $newfile, $oldfile) = split("\t", $line);
+ if ($info =~ /[RC]\d*$/) {
+ # Behold, a rename!
+ # (Or a copy, it's all the same for us.)
+ my $i;
+ for ($i = 0; $i <= $#oldset; $i++) {
+ $oldfile eq $oldset[$i] or next;
+ $rename = 1;
+ splice(@oldset, $i, 1);
+ push(@newset, $newfile);
+ last;
+ }
+ # In case of multiple candidates, follow
+ # all of them:
+ # (TODO: This might be a policy decision
+ # best left on the user.)
+ if ($i > $#oldset and grep { $oldfile eq $_ } @newset) {
+ $rename = 1;
+ push(@newset, $newfile);
+ }
+ } elsif ($info =~ /D$/) {
+ # Not weeding out deleted files might cause bizarre
+ # results when following multiple files since
+ # git-rev-list weeds them out too (probably?).
+ @oldset = grep { $newfile ne $_ } @oldset;
+ @{$head->{'files'}} = grep { $newfile ne $_ } @{$head->{'files'}};
+ }
+ }
+ $head->{'str'} .= "\n";
+delta_loaded:
+
+ if ($rename) {
+ $head->{'recurse'}->{$against} = [@newset, @oldset];
+ }
+ return 1;
+}
+
+sub load_commit($) {
+ my ($head) = @_;
+ $head->{'time'} = undef;
+ $head->{'str'} = '';
+ $head->{'parents'} = ();
+
+ read_commit($head, 0) or return undef;
+
+ # In case there was a merge, the commit will be multiple times
+ # here, each time with a different delta section. Read them all.
+ for (1 .. $#{$head->{'parents'}}) { # stupid vim syntax highlighting
+ read_commit($head, 1) or return undef;
+ }
+
+ return 1;
+}
+
+
+# Add head at the proper position
+sub add_head($) {
+ my ($head) = @_;
+ my $i;
+ for ($i = 0; $i <= $#heads; $i++) {
+ last if ($head->{'time'} > $heads[$i]->{'time'})
+ }
+ splice(@heads, $i, 0, $head);
+}
+
+# Create new head
+sub init_head($@) {
+ my ($rev, @files) = @_;
+ my $head = { files => \@files, 'pipe' => revdiffpipe($rev, @files) };
+ load_commit($head) or return;
+ add_head($head);
+}
+
+
+
+{ # Seed the heads list
+ for my $rev (@revs) {
+ init_head($rev, @files);
+ }
+}
+
+# Process the heads
+{
+ while (@heads) {
+ my $head = splice(@heads, 0, 1);
+
+ print $head->{'str'};
+
+ foreach my $parent (keys %{$head->{'recurse'}}) {
+ init_head($parent, @{$head->{'recurse'}->{$parent}});
+ }
+ $head->{'recurse'} = undef;
+
+ load_commit($head) or next;
+ add_head($head);
+ }
+}
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time. I think
I have forgotten this before.
^ permalink raw reply related
* Fix error handling for nonexistent names
From: Linus Torvalds @ 2006-03-27 0:28 UTC (permalink / raw)
To: Junio C Hamano, Git Mailing List
[ This is an expanded version of a patch I sent out earlier: the
"rev-parse.c" part of it is identical to the earlier version, the
revision.c thing is new ]
When passing in a pathname pattern without the "--" separator on the
command line, we verify that the pathnames in question exist. However,
there were two bugs in that verification:
- git-rev-parse would only check the first pathname, and silently allow
any invalid subsequent pathname, whether it existed or not (which
defeats the purpose of the check, and is also inconsistent with what
git-rev-list actually does)
- git-rev-list (and "git log" etc) would check each filename, but if the
check failed, it would print the error using the first one, ie:
[torvalds@g5 git]$ git log Makefile bad-file
fatal: 'Makefile': No such file or directory
instead of saying that it's 'bad-file' that doesn't exist.
This fixes both bugs.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
diff --git a/rev-parse.c b/rev-parse.c
index 19a5ef7..8ca1c69 100644
--- a/rev-parse.c
+++ b/rev-parse.c
@@ -174,7 +174,9 @@ int main(int argc, char **argv)
char *dotdot;
if (as_is) {
- show_file(arg);
+ if (show_file(arg) && as_is < 2)
+ if (lstat(arg, &st) < 0)
+ die("'%s': %s", arg, strerror(errno));
continue;
}
if (!strcmp(arg,"-n")) {
@@ -194,7 +196,7 @@ int main(int argc, char **argv)
if (*arg == '-') {
if (!strcmp(arg, "--")) {
- as_is = 1;
+ as_is = 2;
/* Pass on the "--" if we show anything but files.. */
if (filter & (DO_FLAGS | DO_REVS))
show_file(arg);
diff --git a/revision.c b/revision.c
index 12cd052..d67718c 100644
--- a/revision.c
+++ b/revision.c
@@ -649,7 +649,7 @@ int setup_revisions(int argc, const char
/* If we didn't have a "--", all filenames must exist */
for (j = i; j < argc; j++) {
if (lstat(argv[j], &st) < 0)
- die("'%s': %s", arg, strerror(errno));
+ die("'%s': %s", argv[j], strerror(errno));
}
revs->prune_data = get_pathspec(revs->prefix, argv + i);
break;
^ permalink raw reply related
* Re: [PATCH] Add git-explode-packs
From: Junio C Hamano @ 2006-03-27 3:53 UTC (permalink / raw)
To: Jan-Benedict Glaw; +Cc: git
In-Reply-To: <20060326125450.GT31387@lug-owl.de>
Jan-Benedict Glaw <jbglaw@lug-owl.de> writes:
> On Sat, 2006-03-25 22:12:46 -0800, Junio C Hamano <junkio@cox.net> wrote:
>> The script seems to do what it claims to, but now why would one
>> need to use this? In other words what's the situation one would
>> find this useful?
>
> It's possibly useful if you oftenly access old objects with
> git-cat-file or git-ls-tree.
Benchmarks?
I created two cloned repositories from git.git. victim03
repository is fully packed with the default pack parameter of
depth and window set both to 10. victim04 repository has the
same set of objects and refs but the pack is expanded (16232
loose objects).
Now in victim03 repository, 657 blobs have depth 10 (i.e. you
need to inflate and apply delta 10 times to get to the object).
So I made the list of these "expensive to access" objects and
run this:
$ cd victim03
$ /usr/bin/time sh -c '
while read sha1; do git cat-file blob $sha1;
done >/dev/null <list
'
3.43user 3.36system 0:07.17elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+364561minor)pagefaults 0swaps
3.51user 3.33system 0:07.10elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+364499minor)pagefaults 0swaps
3.76user 2.99system 0:07.28elapsed 92%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+365155minor)pagefaults 0swaps
With the same file list, in victim04 repository that has 16232
loose objects:
$ cd victim04
$ /usr/bin/time sh -c '
while read sha1; do git cat-file blob $sha1;
done >/dev/null <../victim03/list
'
3.29user 2.98system 0:06.33elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+348786minor)pagefaults 0swaps
3.26user 2.88system 0:06.63elapsed 92%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+347512minor)pagefaults 0swaps
3.16user 2.98system 0:06.20elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+347489minor)pagefaults 0swaps
So you are getting slight performance gain out of this by
exploding the pack, but on the other hand you are taxing the
buffer cache quite heavily by reading the loose objects (in both
of the experiments above, I discarded numbers from the very
first run). The size of object databases in these cases are:
$ du -sh victim0[34]/.git/objects
6.2M victim03/.git/objects
84M victim04/.git/objects
So I am still not convinced it would be useful in general. It
used to be that exploding everything and repacking was the only
way to clean out garbage from packs, but after "repack -a -d"
was invented by Frank Sorenson that became more convenient way.
Especially with the recent "delta reusing" pack-objects, doing
"repack -a -d" has become quite cheap, so...
^ permalink raw reply
* Re: Following renames
From: Marco Costalba @ 2006-03-27 5:47 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jakub Narebski, git
In-Reply-To: <Pine.LNX.4.64.0603261422280.15714@g5.osdl.org>
On 3/27/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
>
> On Sun, 26 Mar 2006, Marco Costalba wrote:
> >
> > FIRST WAY
> >
> > After annotating a file history (double click on a file name in
> > bottom-right window or directly from tree view), you see the whole
> > file annotated. If you have the diff window open you see also the
> > corresponding patch (scrolled to selected file name).
>
> The problem is that this step is already _way_ too expensive.
>
> I don't want to work with any tool that makes "Step 1" take a minute or
> two for a project that has a few years of history. Try it on the linux
> historic project with some file that gets lots of modifications.
>
Historic Linux test (63428 revisions)
File: drivers/net/tg3.c
Revisions that modify tg3.c : 292
With qgit
15s to retrieve file history (git-rev-list)
19.5s to annotate (git-diff-tree -p, current GNU algorithm, not new faster one)
and...
$ time git-whatchanged HEAD drivers/net/tg3.c > /dev/null
98.01user 2.44system 1:46.19elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (797major+43033minor)pagefaults 0swaps
NOTE: It seems that git-whatchanged asks for checked the out file to
work. It didn't work with no repository checked out.
Marco
^ permalink raw reply
* Re: Following renames
From: Paul Jakma @ 2006-03-27 6:00 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <e05354$cc9$1@sea.gmane.org>
On Sun, 26 Mar 2006, Jakub Narebski wrote:
> I think one of the better ideas/suggestions about *recording* filenames was
> in the "impure renames / history tracking" thread
> http://marc.theaimsgroup.com/?l=git&m=114122175216489&w=2
> <Pine.LNX.4.64.0603011343170.13612@sheen.jakma.org>
For the record, the responses I received were educational ;).
Sufficiently so I no longer think renames should be recorded. At
least, definitely not as renames.
I now grok the reasoning for doing it by 'similarity' - it is indeed
a *much* more useful concept. (E.g. the 'pickaxe' idea people keep
alluding though sounds amazingly useful).
So the question really is what, if any, weaknesses does the current
similarity estimation have, and how to solve them. I can think of two
weaknesses:
1. the similarity algorithms can be expensive potentially, and they
essentially get run a lot with the same inputs, to produce the
same results - over and over as one works with a git repo. (there
was a thread a while ago on this I think).
2. Some 'similarities' are just not deducible by current software
state of the art. E.g. where some code is rewritten in another
language:
foo.X -> foo.Y
The high-level algorithms may remain the exact same, but the code
may be unrecognisable as similar except to a human. However,
tracking history back across this rewrite probably would still be
valuable to the human.
So I think what /might/ be interesting is to have a 'similarity
cache', which would help 1, and to allow for manual injection of such
hints (into a seperate and stronger cache most likely) - which would
help 2.
Something to record the following information:
(tree1,tree2)[1]:
Id1 <-> Id1'
.
.
.
Idn <-> Idn'
That would allow:
1. Performance repercussions of similarity estimation to be one-time,
cached there-after. (throw-away information, if a better
similarity estimation heuristic comes along, you can rebuild this
cache)
2. The user to inject their own 'hints' into similarity estimation,
particularly for cases that just aren't obvious and probably never
will be to software estimators (e.g. the rewrite cases), but where
the user sees value in being able to follow back the history.
Avoids:
- encoding anything permanently into the repository (which was
something I was thinking of, and others before me apparently, but
which I now accept would be an awful idea ;) ).
1. I'm not sure if it should be indexed by (commit ID) or
(tree1,tree2) tuple. ??
regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
Fortune:
Men take only their needs into consideration -- never their abilities.
-- Napoleon Bonaparte
^ permalink raw reply
* Re: Following renames
From: Junio C Hamano @ 2006-03-27 6:46 UTC (permalink / raw)
To: Marco Costalba; +Cc: git
In-Reply-To: <e5bfff550603262147t3aec8da6p6bf2a333e2d35f1d@mail.gmail.com>
"Marco Costalba" <mcostalba@gmail.com> writes:
> NOTE: It seems that git-whatchanged asks for checked the out file to
> work. It didn't work with no repository checked out.
Perhaps,
$ git-whatchanged HEAD -- drivers/net/tg3.c
as Linus explained in a separate message today...
^ permalink raw reply
* Re: Following renames
From: Jakub Narebski @ 2006-03-27 6:55 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.64.0603260947100.15714@g5.osdl.org>
Linus Torvalds wrote:
> On Sun, 26 Mar 2006, Jakub Narebski wrote:
>>
>> If (2) is common enough then discussed improvements to rename detection,
>> namely comparing basenames as a base for candidate selection is a good
>> idea.
>
> BK had this "renametool" which got started automatically when you applied
> a patch that removed one or more files and added one or more files, so
> that you could then pair up the files manually.
[...]
> The thing is, the fast rename detection that is in the "next" branch
> really does a lot better, and it's fast enough.
I was thinking about the fast ename detection algorithm in "next" branch.
That is the question if recording additional (helper) information about
contents copying and moving like the mentioned "renametool" did is worth
the effort, both in coding it and from user's point of view. Or would
better contents copying and moving detection ("renames detection") for
whatchanged and similar suffice.
I am of opinion that voluntary information about contents moving and copying
in the commits would help.
Purposes:
1.) Record contents moving and similarity information which cannot or cannot
be easily calculated; see Paul Jakma response in this thread
MessageID: <Pine.LNX.4.64.0603270642090.5276@sheen.jakma.org>
for example copying fragment of code, small fragment of the whole file,
creating documentation or header file from code, or code skeleton from
template, or rewrite of code in different language (e.g. shell script to
perl, script to compiled code e.g. Perl or Python to C).
2.) Caching the results of similarity algorithm/rename detection tool (also
Paul Jakma post), including remembering false positives and undetected
renames, for efficiency. Calculated automatically parts might be
throw-away.
Sources of information:
1.) Manually entered information *at commit*, including *-rm, *-mv, *-cp
like commands (which nobody likes) and systematized (pseudolanguage?) for
copying and moving contents in the log messages.
2.) Semi-manual tools like the mentioned "renametool" of BK.
3.) Support from editor (remebering where copied and pasted, or cut and
pasted fragment came from, and providing prefilled command to record
contents moving ("renames") or prefilled commit log containing this
information. Hard to get, probably most useful.
4.) Information from resolved merges and results of diagnosis (pickaxe like)
tools, especially recording "renames" which were not detected, and removing
"renames" which were detected falsily.
Is that the place where I should provide code (patch) for testing the
idea :) ?
>> I wonder how common is (2) compared to (1)+(2) i.e. move to other dir
>> and rename, old-dir/old-file.c to new-dir/new-subdir/new-file.c
>
> For example, one common case was a directory structure like
>
> ..
> type-file1.c
> type-file2.c
> otherfiles.c
> yet-more.c
> ..
>
> being split up into a subdirectory
>
> ..
> type/file1.c
> type/file2.c
> otherfiles.c
> yet-more.c
> ..
>
> (eg drivers/scsi/aic7xx-* being given a subdirectory of it's own, as
> drivers/scsi/aic7xx/*). So the basename wouldn't stay the same, because it
> contained some piece of data that became redundant with the move.
Perhaps fast rename detection algorithm needs some smart similarity estimate
for names, which would put more weight in the parts closer to basename, and
would detect */type-file1.c and */type/file1.c as similar.
--
Jakub Narebski
Warsaw, Poland
^ permalink raw reply
* Re: Following renames
From: Junio C Hamano @ 2006-03-27 7:30 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Petr Baudis, git
In-Reply-To: <Pine.LNX.4.64.0603261509320.15714@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> No. "--sparse" still removes the uninteresting parents of merges. It just
> doesn't then make the linear history any denser.
Hmph, you are right. add_parents_to_list() calls prune_fn
unconditionally while running limit_list().
Disabling that with yet another flag might be a possibility but
I suspect then it would not be much different from running
rev-list without path limiter and having the caller process the
result.
^ permalink raw reply
* Re: Following renames
From: David Lang @ 2006-03-27 7:40 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <e0827k$7tk$1@sea.gmane.org>
On Mon, 27 Mar 2006, Jakub Narebski wrote:
> 2.) Caching the results of similarity algorithm/rename detection tool (also
> Paul Jakma post), including remembering false positives and undetected
> renames, for efficiency. Calculated automatically parts might be
> throw-away.
this sounds like it could easily devolve into a O(n!) situation where you
are cacheing how everything is related (or not related) to everything
else. Paul was makeing the point that the purpose was to cache the data to
eliminate the time needed to calculate it, but if you don't store all the
results then you don't know if the result is not relavent, or unknown, so
you need to calculate it again.
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
^ permalink raw reply
* Re: Following renames
From: Jakub Narebski @ 2006-03-27 7:53 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.62.0603262337580.26865@qynat.qvtvafvgr.pbz>
David Lang wrote:
> On Mon, 27 Mar 2006, Jakub Narebski wrote:
>
>> 2.) Caching the results of similarity algorithm/rename detection tool
>> (also Paul Jakma post), including remembering false positives and
>> undetected renames, for efficiency. Calculated automatically parts might
>> be throw-away.
>
> this sounds like it could easily devolve into a O(n!) situation where you
> are cacheing how everything is related (or not related) to everything
> else. Paul was makeing the point that the purpose was to cache the data to
> eliminate the time needed to calculate it, but if you don't store all the
> results then you don't know if the result is not relavent, or unknown, so
> you need to calculate it again.
First of all, you only remember non-trivial relations (i.e. file.c is always
related to file.c). If the cache would be only for commits, it would be
O(c*p*n), where c is number of commits, p is percentage of contents moving
("renames") times percent of files changed in the commit, and n is the
number of files, probably O(c) practically. Even if we remember for all
(tree1,tree2) pairs it would be O(c^2). Additionally cache can be limited
in size (pruning oldest cache).
--
Jakub Narebski
Warsaw, Poland
^ permalink raw reply
* Re: Following renames
From: Linus Torvalds @ 2006-03-27 8:07 UTC (permalink / raw)
To: Marco Costalba; +Cc: Jakub Narebski, git
In-Reply-To: <e5bfff550603262147t3aec8da6p6bf2a333e2d35f1d@mail.gmail.com>
On Mon, 27 Mar 2006, Marco Costalba wrote:
>
> Historic Linux test (63428 revisions)
>
> File: drivers/net/tg3.c
> Revisions that modify tg3.c : 292
>
> With qgit
> 15s to retrieve file history (git-rev-list)
> 19.5s to annotate (git-diff-tree -p, current GNU algorithm, not new faster one)
.. and it does absolutely _nothing_ while it's doing that, does it?
> $ time git-whatchanged HEAD drivers/net/tg3.c > /dev/null
> 98.01user 2.44system 1:46.19elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (797major+43033minor)pagefaults 0swaps
In contrast, git-whatchanged will start outputting the recent changes
immediately.
And that's the point. Almost always, we're interested in the _recent_
stuff. The fact that it takes longer to get the old history is not very
important. You generally don't ask "what changed in this file" for a file
that hasn't changed in five years.
Linus
^ permalink raw reply
* Re: Following renames
From: Marco Costalba @ 2006-03-27 11:19 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jakub Narebski, git
In-Reply-To: <Pine.LNX.4.64.0603270005330.15714@g5.osdl.org>
On 3/27/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
>
> On Mon, 27 Mar 2006, Marco Costalba wrote:
> >
> > Historic Linux test (63428 revisions)
> >
> > File: drivers/net/tg3.c
> > Revisions that modify tg3.c : 292
> >
> > With qgit
> > 15s to retrieve file history (git-rev-list)
> > 19.5s to annotate (git-diff-tree -p, current GNU algorithm, not new faster one)
>
> .. and it does absolutely _nothing_ while it's doing that, does it?
>
yes, it's true.
> > $ time git-whatchanged HEAD drivers/net/tg3.c > /dev/null
> > 98.01user 2.44system 1:46.19elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
> > 0inputs+0outputs (797major+43033minor)pagefaults 0swaps
>
> In contrast, git-whatchanged will start outputting the recent changes
> immediately.
>
> And that's the point. Almost always, we're interested in the _recent_
> stuff. The fact that it takes longer to get the old history is not very
> important. You generally don't ask "what changed in this file" for a file
> that hasn't changed in five years.
>
We could run git-rev-list with a time range specifier (changes of last
year as example) by default so to have fast results and run all time
history _only_ on request.
This perhaps could solve the fast output for recent revs problem, if
this is the problem.
I still think the problem with annotation is that you don't see
patches that _remove_ lines of code, you need the whole diff for this.
Marco
^ permalink raw reply
* [PATCH] Reintroduce svn pools to solve the memory leak.
From: Santi Béjar @ 2006-03-27 11:26 UTC (permalink / raw)
To: Jan-Benedict Glaw; +Cc: git, Junio C Hamano
On 3/24/06, Santi Béjar <sbejar@gmail.com> wrote:
> Jan-Benedict Glaw <jbglaw@lug-owl.de> writes:
>
> > On Wed, 2006-03-22 14:33:37 +0100, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
> >
> > Since it seems nobody looked at the GCC import run (which means to use
> > the svnimport), I ran it again, under strace control:
> >
> >> GCC
> >> ~~~
> >> $ /home/jbglaw/bin/git svnimport -C gcc -v svn://gcc.gnu.org/svn/gcc
> >
> >> Committed change 3936:/ 1993-03-31 05:44:03)
> >> Commit ID ceff85145f8671fb2a9d826a761cedc2a507bd1e
> >> Writing to refs/heads/origin
> >> DONE: 3936 origin ceff85145f8671fb2a9d826a761cedc2a507bd1e
> >> ... 3937 trunk/gcc/final.c ...
> >> Can't fork at /home/jbglaw/bin/git-svnimport line 379.
> >
>
> I have the same (?) problem with one of my svn repository. It worked
> before (I've redone the import with the -r flag), so I bisected it.
> The problematic commit seems to be:
>
> diff-tree 4802426... (from 525c0d7...)
> Author: Karl Hasselström <kha@treskal.com>
> Date: Sun Feb 26 06:11:27 2006 +0100
>
> svnimport: Convert executable flag
>
> Convert the svn:executable property to file mode 755 when converting
> an SVN repository to GIT.
>
> Signed-off-by: Karl Hasselström <kha@treskal.com>
> Signed-off-by: Junio C Hamano <junkio@cox.net>
>
> :100755 100755 ee2940f... 6603b96... M git-svnimport.perl
>
> I think it has a memory leak, it used up to 140m of memory.
>
> $ git reset --hard 4802426^
> $ time ../git-svnimport.perl file:///path/
> Use of uninitialized value in string eq at ../git-svnimport.perl line 463.
> Use of uninitialized value in substitution (s///) at ../git-svnimport.perl line 466.
> real 0m55.801s
> user 0m30.578s
> sys 0m23.084s
>
> $ git reset --hard 4802426
> $ time ../git-svnimport.perl file:///path/
> Use of uninitialized value in string eq at ../git-svnimport.perl line 463.
> Use of uninitialized value in substitution (s///) at ../git-svnimport.perl line 466.
> Can't fork at /home/santi/usr/src/scm/git/git-svnimport.perl line 331.
> real 6m2.163s
> user 0m20.332s
> sys 0m50.180s
>
> and it didn't finished. Hope it helps.
And this patch fixes my problems.
---
Introduced in 4802426.
Signed-off-by: Santi Béjar <sbejar@gmail.com>
---
git-svnimport.perl | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/git-svnimport.perl b/git-svnimport.perl
index 639aa41..f2cf062 100755
--- a/git-svnimport.perl
+++ b/git-svnimport.perl
@@ -135,8 +135,10 @@
print "... $rev $path ...\n" if $opt_v;
my (undef, $properties);
+ my $pool = SVN::Pool->new();
eval { (undef, $properties)
- = $self->{'svn'}->get_file($path,$rev,$fh); };
+ = $self->{'svn'}->get_file($path,$rev,$fh,$pool); };
+ $pool->clear;
if($@) {
return undef if $@ =~ /Attempted to get checksum/;
die $@;
^ permalink raw reply related
* Re: Following renames
From: Johannes Schindelin @ 2006-03-27 11:30 UTC (permalink / raw)
To: Marco Costalba; +Cc: git
In-Reply-To: <e5bfff550603270319w20796918wc8f8fe30a6c5627@mail.gmail.com>
Hi,
On Mon, 27 Mar 2006, Marco Costalba wrote:
> I still think the problem with annotation is that you don't see
> patches that _remove_ lines of code, you need the whole diff for this.
Interesting. You'd need a "git-emalb" (blame, but reverse), and you'd need
to tell it a range "rev1..rev2" which is *not* to be interpreted as "^rev1
rev2" but as a direct path from rev1 to rev2.
Ciao,
Dscho
^ permalink raw reply
* Re: Following renames
From: Marco Costalba @ 2006-03-27 11:55 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jakub Narebski, git
In-Reply-To: <Pine.LNX.4.64.0603270005330.15714@g5.osdl.org>
On 3/27/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
> In contrast, git-whatchanged will start outputting the recent changes
> immediately.
>
To integrate git-whatchanged like functionality with filter on a
specific code region, the Linus original request, I am wondering about
something like this:
A new git-diff-tree option --range=a..b to delimit a region,
identified by code lines bounduaries.
As example
git-diff-tree --range=10..15 HEAD -- <path>
Coud give these answers, added to standard git-diff-tree output:
* 10..25 --> modified region new region bounduaries are lines from 10 to 25
15..20 --> region _NOT_ modified but new region bounduaries are
lines from 15 to 20 (perhaps patch added 5 lines _before_ the region)
10..15 ---> region _NOT_ modified and lines, if any, added/removed
_after_ the region
* 10..15 --> modified region with the same boundiaries (as example
removing trailing witespaces)
With this new option of git-diff-tree becames very simple to retrieve
a file history limited to a region, because the region bounduaries in
ouput from one rev are feed as input in parent rev.
Comments?
Marco
^ permalink raw reply
* Re: Following renames
From: Andreas Ericsson @ 2006-03-27 12:27 UTC (permalink / raw)
To: Marco Costalba; +Cc: Linus Torvalds, Jakub Narebski, git
In-Reply-To: <e5bfff550603270355s4b71c306hb4cb2b96eafd0f6e@mail.gmail.com>
Marco Costalba wrote:
> On 3/27/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
>>In contrast, git-whatchanged will start outputting the recent changes
>>immediately.
>>
>
>
> To integrate git-whatchanged like functionality with filter on a
> specific code region, the Linus original request, I am wondering about
> something like this:
>
> A new git-diff-tree option --range=a..b to delimit a region,
> identified by code lines bounduaries.
>
Make it --line-range if you implement this. On a first glance I thought
you meant --commit-range.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: Following renames
From: Linus Torvalds @ 2006-03-27 16:52 UTC (permalink / raw)
To: Marco Costalba; +Cc: Jakub Narebski, git
In-Reply-To: <e5bfff550603270319w20796918wc8f8fe30a6c5627@mail.gmail.com>
On Mon, 27 Mar 2006, Marco Costalba wrote:
> >
> > And that's the point. Almost always, we're interested in the _recent_
> > stuff. The fact that it takes longer to get the old history is not very
> > important. You generally don't ask "what changed in this file" for a file
> > that hasn't changed in five years.
>
> We could run git-rev-list with a time range specifier (changes of last
> year as example) by default so to have fast results and run all time
> history _only_ on request.
Yes.
However, what I've been meaning to do (but just haven't had the time and
energy for so far) is to fix "git-rev-list" with a path limiter.
Right now that always causes things to be totally serialized, and the
revision walking will first look up _all_ the history (well, it will prune
out the merges) before starting to output stuff.
So right now in order for "git-whatchanged" to be fast and incremental, it
doesn't do any path limiting with git-rev-list at ALL, and does it all in
git-diff-tree. Which is horrid.
> I still think the problem with annotation is that you don't see
> patches that _remove_ lines of code, you need the whole diff for this.
Well, that's just another reason "annotate" sucks.
If you select a range of lines, my suggested tool _would_ show you lines
that got removed there, and git-whatchanged does it quite well.
I really think "annotate" is _fundamentally_ a broken operation. It's not
what any sane developer actually wants, and it has serious limitations (ie
it depends on whole history, and it cannot show removals well).
Linus
^ permalink raw reply
* Re: git-svn name
From: Chris Wright @ 2006-03-27 17:48 UTC (permalink / raw)
To: Eric Wong; +Cc: git, Gerrit Pape, Chris Wright
In-Reply-To: <20060326030425.GA6306@hand.yhbt.net>
* Eric Wong (normalperson@yhbt.net) wrote:
> Would distro package maintainers also be willing to add my git-svn
> script to their git-svn binary packages when a new release of git is
> made, too? It's quite different from git-svnimport (see
> contrib/git-svn/git-svn.txt for details).
I think your script name is fine. Best way to handle this is with a
patch to make your git-svn part of the git-svn packaging.
thanks,
-chris
^ permalink raw reply
* Re: [PATCH] Reintroduce svn pools to solve the memory leak.
From: Junio C Hamano @ 2006-03-27 18:16 UTC (permalink / raw)
To: Santi Béjar; +Cc: Jan-Benedict Glaw, git, Karl Hasselström
In-Reply-To: <8aa486160603270326i3a8ddcfau61ca84cdac036ff9@mail.gmail.com>
"Santi Béjar" <sbejar@gmail.com> writes:
> On 3/24/06, Santi Béjar <sbejar@gmail.com> wrote:
>> Jan-Benedict Glaw <jbglaw@lug-owl.de> writes:
>>
>> diff-tree 4802426... (from 525c0d7...)
>> Author: Karl Hasselström <kha@treskal.com>
>> Date: Sun Feb 26 06:11:27 2006 +0100
>>
>> svnimport: Convert executable flag
>>
>> Convert the svn:executable property to file mode 755 when converting
>> an SVN repository to GIT.
>>
>> Signed-off-by: Karl Hasselström <kha@treskal.com>
>> Signed-off-by: Junio C Hamano <junkio@cox.net>
>>
>> :100755 100755 ee2940f... 6603b96... M git-svnimport.perl
>
> And this patch fixes my problems.
Jan-Benedict, thanks for pinpointing the regression, and Santi,
thanks for the patch.
I should have looked a bit more closely when applying the patch
-- it is clear that the patch is doing more than what its log
says. My fault.
Karl, were there other reasons you needed to disable the pool
here (maybe to work around a problem with incompatible version
of SVN module)? I see some other uses of SVN::Pool still there
in the code, so I am assuming this was a simple typo, but just
in case...
^ permalink raw reply
* [PATCH] cogito: Push tags over http
From: Dennis Stosberg @ 2006-03-27 19:12 UTC (permalink / raw)
To: git
A trivial patch for cg-push allows to push tags over http.
Signed-off-by: Dennis Stosberg <dennis@stosberg.net>
---
cg-push | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
e9540d5f524c54102a93570031fb59156cec4188
diff --git a/cg-push b/cg-push
index b6b8954..4332b28 100755
--- a/cg-push
+++ b/cg-push
@@ -70,7 +70,7 @@ sprembranch=":refs/heads/$rembranch"
if [ "${uri#http://}" != "$uri" -o "${uri#https://}" != "$uri" ]; then
# git-http-push doesn't like $sprembranch
- git-http-push "$uri/" "$locbranch:$rembranch"
+ git-http-push "$uri/" "$locbranch:$rembranch" "${tags[@]}"
elif [ "${uri#git+ssh://}" != "$uri" ]; then
send_pack_update "$name" "$(echo "$uri" | sed 's#^git+ssh://\([^/]*\)\(/.*\)$#\1:\2#')" "$locbranch$sprembranch" "${tags[@]}"
elif [ "${uri#rsync://}" != "$uri" ]; then
--
1.2.GIT
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox