Re: [PATCH/RFC 0/3] faster inexact rename handling

["Andy C" <andychup@gmail.com>]

On 10/30/07, Jeff King <peff@peff.net> wrote:
> On Tue, Oct 30, 2007 at 08:38:24AM -0700, Linus Torvalds wrote:
>
> > > with the old and new code. Pairs like Documentation/git-add-script.txt
> > > -> Documentation/git-add.txt are not found, because the file is composed
> > > almost entirely of boilerplate.
> >
> > Ok, that does imply to me that we cannot just drop boilerplate text,
> > because the fact is, lots of files contain boilerplate, but people still
> > think they are "similar".
>
> Well, the problem is that instead of just "dropping" boilerplate text,
> we fail to count it as a similarity, but it still counts towards the
> file size. It may be that just dropping it totally is the right thing
> (in which case those renames _will_ turn up, because they will be filled
> with identical non-boilerplate goodness).

Right, in the demo I make an extra pass after the inverted indexing
step to prune the index -- which means eliminating the common lines
*entirely* from the index (so they don't get attributed to a random
file) *and* decrementing all the file sizes by 1.  That way the
similarity scores shouldn't get skewed.

And as you mentioned we could bump the threshold from 1 to some other
small integer.  Intuitively I guess you could say it is common to copy
a file to 2 places or 3 places, and you don't want all the lines to
get thrown out because of that.  But usually you don't copy a file to
10 or 50 places.

Andy