Directory renames without breaking git log.

Directory renames without breaking git log.

[Kai Blin]

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

Hi folks,

in an effort to make Samba development easier, we're trying to merge the 
Samba3 and Samba4 branches into a single branch. In order to do so, we need 
to rename the "source" directories both Samba 3 and Samba 4 have (we're 
planning to use source3 and source4).

Unfortunately, the directories are big enough that git log stops to track the 
renamed files, so e.g. git log ./samba3 does not show the samba3 history. The 
history is not lost, of course, but it's way less intuitive to get it.

Here's how we merged the two branches:

$ mkdir samba-merged
$ cd samba-merged
$ git init
... (create COPYING, README and other top-level files, git add them)
$ git commit -m "Initial commit of merged samba"
$ git remote add git://git.samba.org/samba.git samba
$ git remote update
$ cp -a ~/samba3/source source3
$ cp -a ~/samba4/source source4
$ git add source3 source4
$ git write-tree
$ echo "merge branches" | git commit-tree <sha1 git write-tree retured> \
      -p <sha1 of the initial commit> \
      -p <sha1 of the current samba3 head> \
      -p <sha1 of the current samba4 head>
$ git reset --hard <sha1 returned by git commit-tree>
$ git log
... history is there as expected
$ git log samba3
... history is just the merge commit
$ git log samba4
... history is just the merge commit

Is there any way to fix this that doesn't involve changing the history with 
git-filter-branch?

Cheers,
Kai

-- 
Kai Blin
WorldForge developer  http://www.worldforge.org/
Wine developer        http://wiki.winehq.org/KaiBlin
Samba team member     http://www.samba.org/samba/team/
--
Will code for cotton.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Directory renames without breaking git log.

[Tarmigan]

On Wed, Sep 3, 2008 at 2:38 PM, Kai Blin <kai@samba.org> wrote:
> Hi folks,
>
> in an effort to make Samba development easier, we're trying to merge the
> Samba3 and Samba4 branches into a single branch. In order to do so, we need
> to rename the "source" directories both Samba 3 and Samba 4 have (we're
> planning to use source3 and source4).
>
> Unfortunately, the directories are big enough that git log stops to track the
> renamed files, so e.g. git log ./samba3 does not show the samba3 history. The
> history is not lost, of course, but it's way less intuitive to get it.

You can try setting diff.renamelimit to 0 in your ~/.gitconfig.  See
Linus's email here for a similar situation in the kernel:
http://lwn.net/Articles/292948/

Thanks,
Tarmigan

Re: Directory renames without breaking git log.

[Tarmigan]

(you're probably better off asking on the whole list, as I don't know much)

On Wed, Sep 3, 2008 at 11:53 PM, Kai Blin <kai@samba.org> wrote:
> On Thursday 04 September 2008 02:16:24 you wrote:
>> On Wed, Sep 3, 2008 at 2:38 PM, Kai Blin <kai@samba.org> wrote:
>> > Unfortunately, the directories are big enough that git log stops to track
>> > the renamed files, so e.g. git log ./samba3 does not show the samba3
>> > history. The history is not lost, of course, but it's way less intuitive
>> > to get it.
>>
>> You can try setting diff.renamelimit to 0 in your ~/.gitconfig.  See
>> Linus's email here for a similar situation in the kernel:
>> http://lwn.net/Articles/292948/
>
> That doesn't seem to fix "git log path/to/file" cases. The really interesting
> part is that if I try git log --follow -M -C path/to/file, I don't get any
> history at all. (--follow is the culprit, if I remove that I at least get the
> merge commit)

OK, I actually tried with your example, and see what you mean.  It
looks like follow does not work with this case.  See
http://permalink.gmane.org/gmane.comp.version-control.git/85766
for an illustration of --follow not working with the subtree merge
strategy, which as far as I can tell is pretty much the same as this
case.  Also it sounds like follow only works with single files, and
not directories:
http://kerneltrap.org/mailarchive/git/2008/5/4/1714694

Why these do not work is way beyond my git knowledge.  As far as I
know (which is very little), follow could work for these cases, but it
currently does not.

> git blame still works, and git log --sparse path/to/file works, of
> course. --sparse makes giving a path a bit pointless, of course, but we
> probably can live with that for time being. I'm still open for suggestions,
> of course. :)

Glad that git blame works.  If you haven't already, you should also
try git gui blame too for better interactivity.

Thanks,
Tarmigan

Re: Directory renames without breaking git log.

[Kai Blin]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

On Thursday 04 September 2008 02:16:24 Tarmigan wrote:
> On Wed, Sep 3, 2008 at 2:38 PM, Kai Blin <kai@samba.org> wrote:
> > Unfortunately, the directories are big enough that git log stops to track
> > the renamed files, so e.g. git log ./samba3 does not show the samba3
> > history. The history is not lost, of course, but it's way less intuitive
> > to get it.
>
> You can try setting diff.renamelimit to 0 in your ~/.gitconfig.  See
> Linus's email here for a similar situation in the kernel:
> http://lwn.net/Articles/292948/

That doesn't seem to fix "git log path/to/file" cases. The really interesting 
part is that if I try git log --follow -M -C path/to/file, I don't get any 
history at all. (--follow is the culprit, if I remove that I at least get the 
merge commit)

git blame still works, and git log --sparse path/to/file works, of 
course. --sparse makes giving a path a bit pointless, of course, but we 
probably can live with that for time being. I'm still open for suggestions, 
of course. :)

Cheers,
Kai

-- 
Kai Blin
WorldForge developer  http://www.worldforge.org/
Wine developer        http://wiki.winehq.org/KaiBlin
Samba team member     http://www.samba.org/samba/team/
--
Will code for cotton.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Directory renames without breaking git log.

[Junio C Hamano]

Kai Blin <kai@samba.org> writes:

> On Thursday 04 September 2008 02:16:24 Tarmigan wrote:
>> On Wed, Sep 3, 2008 at 2:38 PM, Kai Blin <kai@samba.org> wrote:
>> > Unfortunately, the directories are big enough that git log stops to track
>> > the renamed files, so e.g. git log ./samba3 does not show the samba3
>> > history. The history is not lost, of course, but it's way less intuitive
>> > to get it.
>>
>> You can try setting diff.renamelimit to 0 in your ~/.gitconfig.  See
>> Linus's email here for a similar situation in the kernel:
>> http://lwn.net/Articles/292948/
>
> That doesn't seem to fix "git log path/to/file" cases. The really interesting 
> part is that if I try git log --follow -M -C path/to/file, I don't get any 
> history at all. (--follow is the culprit, if I remove that I at least get the 
> merge commit)
>
> git blame still works, and git log --sparse path/to/file works, of 
> course. --sparse makes giving a path a bit pointless, of course, but we 
> probably can live with that for time being. I'm still open for suggestions, 
> of course. :)

Give both directories, like:

	"git log -- newdir olddir"

perhaps?

Re: Directory renames without breaking git log.

[Kai Blin]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Thursday 04 September 2008 21:49:27 Junio C Hamano wrote:

> > git blame still works, and git log --sparse path/to/file works, of
> > course. --sparse makes giving a path a bit pointless, of course, but we
> > probably can live with that for time being. I'm still open for
> > suggestions, of course. :)
>
> Give both directories, like:
>
> 	"git log -- newdir olddir"
>
> perhaps?

Better, but really ugly, as we'll have to keep doing this for the rest of the 
project's life to get the full history. And while it's all nice and fun for 
git log -- source3/configure.in source/configure.in, it's less fun for deeper 
paths.

We'll probably end up just doing a git-filter-branch renaming the samba3 
source dir source3 from the beginning and the samba4 source dir source4 from 
the beginning, and then do the octopus merge. Without any paths changing, 
that should probably work. It's a bit annoying to break all external 
branches, but we only need to do this once, and people will only need to 
git-format-patch and git-am once, and we can provide a step-by-step guide for 
this as well.

Thanks for the feedback, though. :)

Cheers,
Kai

-- 
Kai Blin
WorldForge developer  http://www.worldforge.org/
Wine developer        http://wiki.winehq.org/KaiBlin
Samba team member     http://www.samba.org/samba/team/
--
Will code for cotton.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Directory renames without breaking git log.

[Junio C Hamano]

Kai Blin <kai@samba.org> writes:

> On Thursday 04 September 2008 21:49:27 Junio C Hamano wrote:
>
>> > git blame still works, and git log --sparse path/to/file works, of
>> > course. --sparse makes giving a path a bit pointless, of course, but we
>> > probably can live with that for time being. I'm still open for
>> > suggestions, of course. :)
>>
>> Give both directories, like:
>>
>> 	"git log -- newdir olddir"
>>
>> perhaps?
>
> Better, but really ugly, as we'll have to keep doing this for the rest
> of the project's life to get the full history. And while it's all nice
> and fun for git log -- source3/configure.in source/configure.in, it's
> less fun for deeper paths.

Yes, following across subtree merges _could_ be improved.

But another thing I should mention in this context is that you should not
take --follow option (at least in the current form) too seriously.

I see it's been a while --- the last time I did this was October 2006 if I
am not mistaken.  It's time of the year I should point at one of the most
important articles ever written on this mailing list:

    http://thread.gmane.org/gmane.comp.version-control.git/27/focus=217

After understanding what Linus envisioned back then, why what he said are
important are important and why what he dismissed as uninteresting are
indeed uninteresting, things to think about now are:

 - "blame" (especially with -C -C), as you found out already, does answer
   the more important question "where did this come from"; and

 - the question "log --follow $this_file" is asking is exactly "where did
   this file come from".  Remember what adjective was used for the
   question in the article?

I also should mention that --follow was done by Linus as a hack with
known limitations.

Potential improvements to follow possible renames fully would involve:

 * allow not just a single path but a set of pathspecs to be recorded
   during --follow traversal;

 * allow the above information be associated with individual commits, not
   as a single global state in the traversal machinery;

 * enhance the logic to update the pathspecs information kept above when
   you hit renames while traversing the history.  An important part of
   this job involves inferring a wholesale rename of a directory by
   looking at many files moved from one place to another, which we
   currently do not do anywhere in git.

If you do all of the above, it will become a feature, not a hack with
known limitation.  But I do not think anybody so far thought it is worth
the effort, only to answer the "where did this file come from" (ituot ue
zaf m eqzeunxq cgqefuaz).

Personally I feel that this is slightly closer to "I might do this myself
if I had infinite amount of time and I am really bored" than "I don't
care; patches welcome" category.

Re: Directory renames without breaking git log.

[Jakub Narebski]

Kai Blin <kai@samba.org> writes:

> On Thursday 04 September 2008 02:16:24 Tarmigan wrote:
> > On Wed, Sep 3, 2008 at 2:38 PM, Kai Blin <kai@samba.org> wrote:
> > > Unfortunately, the directories are big enough that git log stops to track
> > > the renamed files, so e.g. git log ./samba3 does not show the samba3
> > > history. The history is not lost, of course, but it's way less intuitive
> > > to get it.
> >
> > You can try setting diff.renamelimit to 0 in your ~/.gitconfig.  See
> > Linus's email here for a similar situation in the kernel:
> > http://lwn.net/Articles/292948/
> 
> That doesn't seem to fix "git log path/to/file" cases. The really
> interesting part is that if I try "git log --follow -M -C
> path/to/file", I don't get any history at all. (--follow is the
> culprit, if I remove that I at least get the merge commit)
> 
> git blame still works, and git log --sparse path/to/file works, of 
> course. --sparse makes giving a path a bit pointless, of course, but we 
> probably can live with that for time being. I'm still open for suggestions, 
> of course. :)

Unfortunately "git log --follow <filename>" works correctly only
on relative simple histories.  You are of course welcome to improve
this part of git.

Simple workaround is to use "git log <file>" (optionally using
--diff-filter) to get when file vanished, check using "git show" or
"git whatchanged" on boundary commit, then use 
"git log -C -C -- <old name> <new name>"
-- 
Jakub Narebski
Poland
ShadeHawk on #git