how to track the history of a line in a file

how to track the history of a line in a file

[david]

I have a need to setup a repository where I'm storing config files, and I 
need to be able to search the history of a particular line, not just when 
the last edit of the line was (which is what I see from git blame)

I'm not seeing a obvious way to do this, am I missing something or does it 
need a non-obvious approach?

for example, if I do

git blame -L /SUBLEVEL/,+1 -M Makefile

on the linux kernel it currently shows

57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28

what I would want it to show would be a list of the commits that have 
changed this line.

It looks like I can write a script to do this

git blame -L /SUBLEVEL/,+1 -M Makefile 57f8f7b6^
6e86841d (Linus Torvalds 2008-07-28 19:40:31 -0700 3) SUBLEVEL = 27
git blame -L /SUBLEVEL/,+1 -M Makefile 6e86841d^
2ddcca36 (Linus Torvalds 2008-05-03 11:59:44 -0700 3) SUBLEVEL = 26

etc.

is there a better way to do this?

David Lang

Re: how to track the history of a line in a file

[Jeff King]

On Fri, Jan 02, 2009 at 02:13:32PM -0800, david@lang.hm wrote:

> I have a need to setup a repository where I'm storing config files, and I  
> need to be able to search the history of a particular line, not just when  
> the last edit of the line was (which is what I see from git blame)

As you figured out, the "manual" way is to just keep reblaming from the
parent of each blame. Recent versions of "git gui blame" have a "reblame
from parent" option in the context menu which makes this a lot less
painful.

> 57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28
>
> what I would want it to show would be a list of the commits that have  
> changed this line.

The tricky thing here is what is "this line"? Using the line number
isn't right, since it will change based on other content coming in and
out of the file. You can keep drilling down by reblaming parent commits,
but remember that each time you do that you are manually looking at the
content and saying "Oh, this is the line I am still interested in." So I
a script would have to correlate the old version and new version of the
line and realize how to follow the "interesting" thing.

In your case, I think you want to see any commit in Makefile which
changed a line with SUBLEVEL in it. Which is maybe easiest done as:

  git log -z -p Makefile |
    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
    tr '\0' '\n'

and is pretty fast. But obviously we're leveraging some content-specific
knowledge about what's in the Makefile.

-Peff

Re: how to track the history of a line in a file

[david]

On Fri, 2 Jan 2009, Jeff King wrote:

> On Fri, Jan 02, 2009 at 02:13:32PM -0800, david@lang.hm wrote:
>
>> I have a need to setup a repository where I'm storing config files, and I
>> need to be able to search the history of a particular line, not just when
>> the last edit of the line was (which is what I see from git blame)
>
> As you figured out, the "manual" way is to just keep reblaming from the
> parent of each blame. Recent versions of "git gui blame" have a "reblame
> from parent" option in the context menu which makes this a lot less
> painful.

unfortunantly I am needing to do this from the command line.

>> 57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28
>>
>> what I would want it to show would be a list of the commits that have
>> changed this line.
>
> The tricky thing here is what is "this line"? Using the line number
> isn't right, since it will change based on other content coming in and
> out of the file. You can keep drilling down by reblaming parent commits,
> but remember that each time you do that you are manually looking at the
> content and saying "Oh, this is the line I am still interested in." So I
> a script would have to correlate the old version and new version of the
> line and realize how to follow the "interesting" thing.
>
> In your case, I think you want to see any commit in Makefile which
> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>
>  git log -z -p Makefile |
>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>    tr '\0' '\n'
>
> and is pretty fast. But obviously we're leveraging some content-specific
> knowledge about what's in the Makefile.

using the line number shouldn't be _that_ hard becouse git knows what 
lines came and went from the file, so it can calculate the new line number 
(and does with the -M option)

In my case I would consider 'the same line' to be any lines in the diff 
that were taken out when this line was put in

so in the usual case (for me) of

-oldline
+newline

it's a 1-1 correspondence

if it's instead
-oldline1
-oldline2
+newline1
+newline2

I can't know for sure which oldline corresponds to the newline, but the 
odds are very good that they are related, so if I widen the search to 
cover each of the lines I am probably good.

David Lang

Re: how to track the history of a line in a file

[Jeff Whiteside]

> -oldline
> +newline
>
> it's a 1-1 correspondence
>
> if it's instead
> -oldline1
> -oldline2
> +newline1
> +newline2

what a neat idea.  i'm going to start malloc'ing 10,000 lines x 120
chars for each file i add, and edit them so that no new lines replace
removed lines unless it's intended that they were the same line.

Re: how to track the history of a line in a file

[david]

On Fri, 2 Jan 2009, Jeff Whiteside wrote:

>> -oldline
>> +newline
>>
>> it's a 1-1 correspondence
>>
>> if it's instead
>> -oldline1
>> -oldline2
>> +newline1
>> +newline2
>
> what a neat idea.  i'm going to start malloc'ing 10,000 lines x 120
> chars for each file i add, and edit them so that no new lines replace
> removed lines unless it's intended that they were the same line.

I don't understand your comment. that isn't nessasary to do the tracking 
that I'm needing (you don't have to look at every line, only some of the 
lines that are in the same hunk of the patch as the line(s) you are 
interested in)

in my situation the use case is config files. in them each line is usually 
edited independantly of other lines (with stuff being added, usually, but 
not always on the end)

David Lang

Re: how to track the history of a line in a file

[david]

On Fri, 2 Jan 2009, Jeff King wrote:

> The tricky thing here is what is "this line"? Using the line number
> isn't right, since it will change based on other content coming in and
> out of the file. You can keep drilling down by reblaming parent commits,
> but remember that each time you do that you are manually looking at the
> content and saying "Oh, this is the line I am still interested in." So I
> a script would have to correlate the old version and new version of the
> line and realize how to follow the "interesting" thing.
>
> In your case, I think you want to see any commit in Makefile which
> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>
>  git log -z -p Makefile |
>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>    tr '\0' '\n'
>
> and is pretty fast. But obviously we're leveraging some content-specific
> knowledge about what's in the Makefile.

Ok, I hacked togeather a quick bash script to try this

#!/bin/bash
line=`git blame -L /$1/,+1 -M $2`
COMMIT=`echo $line |cut -f 1 -d " "`
foundline=`echo $line |cut -f 6 -d " "|sed s/")"//`
echo "$foundline $COMMIT"
echo "$line"
while [ "$COMMIT" != "" ] ;do
echo "git blame -L $foundline,+1 -M $2 $COMMIT^"
line=`git blame -L $foundline,+1 -M $2 $COMMIT^`
COMMIT=`echo $line |cut -f 1 -d " "`
foundline=`echo $line |cut -f 6 -d " "|sed s/")"//`
echo "$line"
done


the problem that this has is that line 3 of $COMMIT may not be line 3 of 
$COMMIT^, and if they aren't it ends up hunting down the wrong data

either that or I am not understanding the output of git blame properly 
(also very possible)

David Lang

Re: how to track the history of a line in a file

[david]

On Fri, 2 Jan 2009, david@lang.hm wrote:

> On Fri, 2 Jan 2009, Jeff King wrote:
>
>> The tricky thing here is what is "this line"? Using the line number
>> isn't right, since it will change based on other content coming in and
>> out of the file. You can keep drilling down by reblaming parent commits,
>> but remember that each time you do that you are manually looking at the
>> content and saying "Oh, this is the line I am still interested in." So I
>> a script would have to correlate the old version and new version of the
>> line and realize how to follow the "interesting" thing.
>> 
>> In your case, I think you want to see any commit in Makefile which
>> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>>
>>  git log -z -p Makefile |
>>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>>    tr '\0' '\n'
>> 
>> and is pretty fast. But obviously we're leveraging some content-specific
>> knowledge about what's in the Makefile.
>
> Ok, I hacked togeather a quick bash script to try this
>
<SNIP>
> the problem that this has is that line 3 of $COMMIT may not be line 3 of 
> $COMMIT^, and if they aren't it ends up hunting down the wrong data
>
> either that or I am not understanding the output of git blame properly (also 
> very possible)

I was misunderstanding git blame

new script is

#!/bin/bash
line=`git blame -n -b -l -L /$1/,+1 -M $2`
echo "-$line"
foundCOMMIT=`echo "$line" |cut -c -40`
foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "`
while [ "$foundCOMMIT" != "                                        " ] ;do
#git diff -U0 $foundCOMMIT..$foundCOMMIT^ $2
line=`git blame -n -b -l -L $foundline,+1 -M $2 $foundCOMMIT^`
echo "-$line"
foundCOMMIT=`echo "$line" |cut -c -40`
foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "`
done

this seems to be working for me now.

David Lang

Re: how to track the history of a line in a file

[Junio C Hamano]

david@lang.hm writes:

> I was misunderstanding git blame

You may find the --porcelain output format of git-blame useful to make
your script safer and more robust.

Re: how to track the history of a line in a file

[david]

On Fri, 2 Jan 2009, Junio C Hamano wrote:

> david@lang.hm writes:
>
>> I was misunderstanding git blame
>
> You may find the --porcelain output format of git-blame useful to make
> your script safer and more robust.

where is this output defined? I can look at it and understand it, but I 
don't know what is guarenteeded to exist, what will be on a specific line, 
and what may vanish on me.

David Lang

Re: how to track the history of a line in a file

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]

david@lang.hm wrote:
> On Fri, 2 Jan 2009, Junio C Hamano wrote:
> > You may find the --porcelain output format of git-blame useful to make
> > your script safer and more robust.
> 
> where is this output defined? I can look at it and understand it, but I 
> don't know what is guarenteeded to exist, what will be on a specific line, 
> and what may vanish on me.

man git-blame | less -p PORCELAIN\ FORMAT

-- 
Thomas Rast
trast@{inf,student}.ethz.ch


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: how to track the history of a line in a file

[david]

On Sat, 3 Jan 2009, Thomas Rast wrote:

> david@lang.hm wrote:
>> On Fri, 2 Jan 2009, Junio C Hamano wrote:
>>> You may find the --porcelain output format of git-blame useful to make
>>> your script safer and more robust.
>>
>> where is this output defined? I can look at it and understand it, but I
>> don't know what is guarenteeded to exist, what will be on a specific line,
>> and what may vanish on me.
>
> man git-blame | less -p PORCELAIN\ FORMAT

thanks,

am I missing something else or will git blame with a regex only return 
info on the first line it matches?

David Lang

Re: how to track the history of a line in a file

[david]

On Fri, 2 Jan 2009, david@lang.hm wrote:

> On Fri, 2 Jan 2009, david@lang.hm wrote:
>
>> On Fri, 2 Jan 2009, Jeff King wrote:
>> 
>>> The tricky thing here is what is "this line"? Using the line number
>>> isn't right, since it will change based on other content coming in and
>>> out of the file. You can keep drilling down by reblaming parent commits,
>>> but remember that each time you do that you are manually looking at the
>>> content and saying "Oh, this is the line I am still interested in." So I
>>> a script would have to correlate the old version and new version of the
>>> line and realize how to follow the "interesting" thing.
>>> 
>>> In your case, I think you want to see any commit in Makefile which
>>> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>>>
>>>  git log -z -p Makefile |
>>>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>>>    tr '\0' '\n'
>>> 
>>> and is pretty fast. But obviously we're leveraging some content-specific
>>> knowledge about what's in the Makefile.
>> 
>> Ok, I hacked togeather a quick bash script to try this
>> 
> <SNIP>
>> the problem that this has is that line 3 of $COMMIT may not be line 3 of 
>> $COMMIT^, and if they aren't it ends up hunting down the wrong data
>> 
>> either that or I am not understanding the output of git blame properly 
>> (also very possible)
>
> I was misunderstanding git blame
>
> new script is
>
> #!/bin/bash
> line=`git blame -n -b -l -L /$1/,+1 -M $2`
> echo "-$line"
> foundCOMMIT=`echo "$line" |cut -c -40`
> foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "`
> while [ "$foundCOMMIT" != "                                        " ] ;do
> #git diff -U0 $foundCOMMIT..$foundCOMMIT^ $2
> line=`git blame -n -b -l -L $foundline,+1 -M $2 $foundCOMMIT^`
> echo "-$line"
> foundCOMMIT=`echo "$line" |cut -c -40`
> foundline=`echo "$line" |cut -c 42- |cut -f 1 -d " "`
> done
>
> this seems to be working for me now.

not quite, it works as long as the line doesn't move in the commit where 
it changes.

David Lang

Re: how to track the history of a line in a file

[david]

On Fri, 2 Jan 2009, david@lang.hm wrote:

> On Fri, 2 Jan 2009, david@lang.hm wrote:
>
>> On Fri, 2 Jan 2009, david@lang.hm wrote:
>> 
>>> On Fri, 2 Jan 2009, Jeff King wrote:
>>> 
>>>> The tricky thing here is what is "this line"? Using the line number
>>>> isn't right, since it will change based on other content coming in and
>>>> out of the file. You can keep drilling down by reblaming parent commits,
>>>> but remember that each time you do that you are manually looking at the
>>>> content and saying "Oh, this is the line I am still interested in." So I
>>>> a script would have to correlate the old version and new version of the
>>>> line and realize how to follow the "interesting" thing.
>>>> 
>>>> In your case, I think you want to see any commit in Makefile which
>>>> changed a line with SUBLEVEL in it. Which is maybe easiest done as:
>>>>
>>>>  git log -z -p Makefile |
>>>>    perl -0ne 'print if /\n[+-]SUBLEVEL/' |
>>>>    tr '\0' '\n'
>>>> 
>>>> and is pretty fast. But obviously we're leveraging some content-specific
>>>> knowledge about what's in the Makefile.
>>> 
>>> Ok, I hacked togeather a quick bash script to try this
>>> 
>> <SNIP>
>>> the problem that this has is that line 3 of $COMMIT may not be line 3 of 
>>> $COMMIT^, and if they aren't it ends up hunting down the wrong data

solving this problem was a bit more work than expected. I ended up 
invoking git diff to do the step back and find what lines were replaced by 
the one I care about.

this still can run into problems if the lines that are examined are not 
unique (git blame only finds the first match)



#!/usr/bin/perl
# program to trace the history of a line through a git repository
#
# useage findlines pattern filespec
#
# finds all occurances of the pattern in the file and then uses git blame
#  to find where those lines were introduced. It then takes a step back 
and
#  looks at what the line replaced (if anything) and does a git blame on 
those
#  lines as well
#
# written by David Lang and released under GPLv2
#
$target = shift @ARGV;
$file = shift @ARGV;
open($infile,"<$file");
$linecount=0;
# find all lines in the file that match the target pattern
# we have been provided. add them to a stack of items to research
while(<$infile>){
   $linecount++;
   if ($_ =~ /$target/) {
     $items[++$#items]{line} = $linecount;
     $items[$#items]{tree} = "HEAD";
     chomp;
     $items[$#items]{target} = $_;
   }
}
# go through all the items and research them in turn
#  (note that more items may be added to the list while inside this loop)
for ($i=0;$i<=$#items;$i++) {
# do a git blame to find where the lines came from
   $result = `git blame -b -l -L "$items[$i]{line}",+1 -M  $file 
$items[$i]{tree}` ;
   print "in ".$items[$i]{tree}." found:\n$result";
   $commit=substr($result,0,40);
   # if this is not the root commit,
   #  do a diff to find what lines may have been replaced by the line we 
are looking for
   if($commit ne "                                        "){
     $diff=`git diff -U0 $commit^..$commit $file`;
     @lines=split("\n",$diff);
     #strip the first four lines from the diff (as they just talk about the 
file)
     shift(@lines);
     shift(@lines);
     shift(@lines);
     shift(@lines);
     $match=0;
     $hunk='';
     # find what hunk of the diff introduced the lines we are looking for,
     #  add the lines that they replaced to the list of items to examine
     foreach(@lines){
       if ($_ =~ /^\@\@/ ) {
         if ($match) {
           process_hunk();
         } else {
           $hunk = '';
         };
       } else {
         $hunk .= $_."\n";
         if ($_ =~ /$items[$i]{target}/) {
           $match = 1;
         }
       }
     }
     if ($match) {
       process_hunk();
     }
   }
}
sub process_hunk(){
   # find lines in the hunk of diff that are being replaced
   foreach (split("\n",$hunk)){
     if ($_ =~ /^-/ ){
       # for now queue the line contents for further processing.
       #   this really should be the line number becouse otherwise
       #   git blame will use the first match it finds. since we are
       # matching the entire line this is less of a problem than it could 
be.
       $items[++$#items]{line} = "/".substr($_,1)."/";
       $items[$#items]{target} = substr($_,1);
       $items[$#items]{tree} = "$commit^";
     }
   }
}

Re: how to track the history of a line in a file

[Bernt Hansen]

david@lang.hm writes:

> I have a need to setup a repository where I'm storing config files,
> and I need to be able to search the history of a particular line, not
> just when the last edit of the line was (which is what I see from git
> blame)
>
> I'm not seeing a obvious way to do this, am I missing something or
> does it need a non-obvious approach?
>
> for example, if I do
>
> git blame -L /SUBLEVEL/,+1 -M Makefile
>
> on the linux kernel it currently shows
>
> 57f8f7b6 (Linus Torvalds 2008-10-23 20:06:52 -0700 3) SUBLEVEL = 28
>
> what I would want it to show would be a list of the commits that have
> changed this line.
>
> It looks like I can write a script to do this
>
> git blame -L /SUBLEVEL/,+1 -M Makefile 57f8f7b6^
> 6e86841d (Linus Torvalds 2008-07-28 19:40:31 -0700 3) SUBLEVEL = 27
> git blame -L /SUBLEVEL/,+1 -M Makefile 6e86841d^
> 2ddcca36 (Linus Torvalds 2008-05-03 11:59:44 -0700 3) SUBLEVEL = 26
>
> etc.
>
> is there a better way to do this?
>
> David Lang

I think you need a script to do what you want.  I think this works...

Save the following script in ~/bin/git-rblame.sh, make it executable,
and then create a global git alias for it as follows:

$ git config --global alias.rblame '!~/bin/git-rblame.sh $*'

Then you can just use

$ git rblame -L /SUBLEVEL/,+1 -M Makefile
6e86841d (Linus Torvalds 2008-07-28 19:40:31 -0700 3) SUBLEVEL = 27
2ddcca36 (Linus Torvalds 2008-05-03 11:59:44 -0700 3) SUBLEVEL = 26
...
4c91aedb (Linus Torvalds 2005-06-28 22:57:29 -0700 3) SUBLEVEL = 13
^1da177e (Linus Torvalds 2005-04-16 15:20:36 -0700 3) SUBLEVEL = 12

------cut here ------ !/bin/git-rblame.sh ---
#!/bin/sh
PARAMS="$*"
LINE=$(git blame $PARAMS)
while test $? == 0
do
    echo $LINE
    COMMIT="${LINE:0:8}^"
    LINE=$(git blame $PARAMS $COMMIT 2>/dev/null)
done
------cut here ------------------------------

Cheers,
Bernt

Re: how to track the history of a line in a file

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

On Tue, Jan 06, 2009 at 10:48:10AM -0500, Bernt Hansen <bernt@norang.ca> wrote:
> Save the following script in ~/bin/git-rblame.sh, make it executable,
> and then create a global git alias for it as follows:
> 
> $ git config --global alias.rblame '!~/bin/git-rblame.sh $*'

Given that you have ~/bin in PATH, just name the script ~/bin/git-rblame
and you won't even have to define an alias on each machine. ;-)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: how to track the history of a line in a file

[Bernt Hansen]

Miklos Vajna <vmiklos@frugalware.org> writes:

> On Tue, Jan 06, 2009 at 10:48:10AM -0500, Bernt Hansen <bernt@norang.ca> wrote:
>> Save the following script in ~/bin/git-rblame.sh, make it executable,
>> and then create a global git alias for it as follows:
>> 
>> $ git config --global alias.rblame '!~/bin/git-rblame.sh $*'
>
> Given that you have ~/bin in PATH, just name the script ~/bin/git-rblame
> and you won't even have to define an alias on each machine. ;-)

Yes but I don't want to use 'git-rblame' as the command since I've
broken my habit of using the dashed versions of commands. 'git rblame'
just feels better to me.

Now I probably should have named the script something that won't ever
clash with possible future git commands (like my-git-rblame.sh or
something) but since it's in my ~/bin I'll just deal with that if it
ever happens in the future :)

-Bernt

Re: how to track the history of a line in a file

[Bernt Hansen]

Bernt Hansen <bernt@norang.ca> writes:

> Miklos Vajna <vmiklos@frugalware.org> writes:
>
>> On Tue, Jan 06, 2009 at 10:48:10AM -0500, Bernt Hansen <bernt@norang.ca> wrote:
>>> Save the following script in ~/bin/git-rblame.sh, make it executable,
>>> and then create a global git alias for it as follows:
>>> 
>>> $ git config --global alias.rblame '!~/bin/git-rblame.sh $*'
>>
>> Given that you have ~/bin in PATH, just name the script ~/bin/git-rblame
>> and you won't even have to define an alias on each machine. ;-)
>
> Yes but I don't want to use 'git-rblame' as the command since I've
> broken my habit of using the dashed versions of commands. 'git rblame'
> just feels better to me.
>
> Now I probably should have named the script something that won't ever
> clash with possible future git commands (like my-git-rblame.sh or
> something) but since it's in my ~/bin I'll just deal with that if it
> ever happens in the future :)

Oops. That actually does do what I want.  Thanks for pointing that out
Miklos!

(sorry for replying to my own post - next time I'll try it first before
 I post :)

-Bernt