git merge questions

git merge questions

[Don Zickus]

I now have a git merge question.  I decided to try merging branches
and was left with a situation in which the file was moved on the
master branch and updated on my branch.  Not sure how to properly
integrate the changes from my branch.

For example:
%git checkout -b test v2.6.14
%<manually apply all the 2.6.14.z stable patches>
%<commit those patches>
%git checkout -b test2 v2.6.15-rc4
%git pull . test

Now over the course of 2.6.15 the arch/ppc64 was renamed to
arch/powerpc.  Fine. The merge algorithms handled all the unchanged
files properly.  However arch/ppc64/Kconfig was modified and the
merging was left unresolved.  In fact there is no file no merge
against (because it moved).  So 'git-ls-files -u' only shows stage 1+3
(no stage 2, of course).

How do I merge those changes?  I don't know enough about all the git
commands to figure this out, especially how to take advantage of the
stage 1, 2, and 3 files.

Thanks for any help,
Don

Re: git merge questions

[Junio C Hamano]

Don Zickus <dzickus@gmail.com> writes:

> Now over the course of 2.6.15 the arch/ppc64 was renamed to
> arch/powerpc.  Fine. The merge algorithms handled all the unchanged
> files properly.  However arch/ppc64/Kconfig was modified and the
> merging was left unresolved.  In fact there is no file no merge
> against (because it moved).  So 'git-ls-files -u' only shows stage 1+3
> (no stage 2, of course).
>
> How do I merge those changes?  I don't know enough about all the git
> commands to figure this out, especially how to take advantage of the
> stage 1, 2, and 3 files.

I do not work on the kernel myself so I'll probably get the
details wrong, but I am guessing here is what you have:

 - 2.6.14.z has arch/ppc64/Kconfig;

 - Your "test" branch has arch/ppc64/Kconfig.  You may or may
   not have changes since 2.6.14.z on that file;

 - "test2" branch has 2.6.15-rc4, which has the file at
   arch/powerpc/Kconfig;

Then, with the current branch being "test2", you pulled "test"
into it.

 - Untouched [by whom???] files from arch/{ppc64,powerpc}/ were
   merged correctly and you find the result of merge under
   arch/powerpc --- after all you are applying changes on top of
   2.6.15-rc4 and you want them under arch/powerpc as the
   upstream has;

 - arch/ppc64/Kconfig was modified [by whom???] and merge was
   left unresolved.

I suspect these:

	ls -l arch/ppc64/Kconfig <1>
	git ls-files -s arch/ppc64/Kconfig <2>
        ls -l arch/powerpc/Kconfig arch/powerpc/Kconfig~* <3>
        git ls-files -s arch/powerpc/Kconfig <4>

<1> has the file from your "test" branch based on 2.6.14z;
<2> shows that same file;
<3> may show the three variants (I am not sure about this);
<4> has 2.6.15-rc4 version at stage 3 and merge base version at
    stage 1 (I am not absolutely certain what stage 1 has, and
    also am puzzled why stage 2 is not there --- if the
    recursive strategy figured out renames it should have the
    contents of arch/ppc64/Kconfig from test branch there).

The safest (however most primitive) way I can think of to go
from here is:

1. figure out the merge-base Kconfig file's contents.

        $ mb=$(git merge-base refs/heads/test refs/heads/test2)
        $ git ls-tree $(mb) arch/powerpc/Kconfig arch/ppc/Kconfig

   I do not know which path the merge base has, but it should
   have either one of those.  Note the blob object name (call it
   $oSHA1).

2. extract three versions.

	$ git ls-tree refs/heads/test2 arch/powerpc/Kconfig
        $ git ls-tree refs/heads/test arch/powerpc/Kconfig

   Note the blob object name from these two as well (call them
   $aSHA1, and $bSHA1).

3. merge those three.

	$ git cat-file blob $oSHA1 >orig
        $ git cat-file blob $aSHA1 >from-test2
        $ git cat-file blob $bSHA1 >from-test
        $ merge from-test2 orig from-test

With luck from-test2 would contain a clean automerge result, or
you will get <<< === >>> conflict markers.  Resolve them in the
file, and then:

4. resolve the index and remove cruft.

	$ cat from-test2 >arch/powerpc/Kconfig
        $ rm -f arch/ppc64/Kconfig orig from-test2 from-test
        $ git update-index --add --remove \
        	arch/ppc64/Kconfig arch/powerpc/Kconfig

5. if there is no other merge cruft, commit.

	$ git commit

What puzzles me is that I think it is supposed to have done all
the above for you.  Namely:

	$ ls -l arch/ppc64/Kconfig <1>
	$ git ls-files -s arch/ppc64/Kconfig <2>
        $ ls -l arch/powerpc/Kconfig arch/powerpc/Kconfig~* <3>
        $ git ls-files -s arch/powerpc/Kconfig <4>

<1> should not remain --- recursive merge could have notied that
the file was moved.
<2> ditto.
<3> arch/powerpc/Kconfig should be there with possibly merge
conflict markers.
<4> stage1 with merge base version (possibly renamed), stage2
with test2 version, stage3 with your test version (definitely
renamed).

and after editing arch/powerpc/Kconfig to resolve conflicts, you
should be able to just say:

	$ git update-index arch/powerpc/Kconfig
	$ git commit

Re: git merge questions

[Don Zickus]

On 12/16/05, Junio C Hamano <junkio@cox.net> wrote:
> Don Zickus <dzickus@gmail.com> writes:
>
> > Now over the course of 2.6.15 the arch/ppc64 was renamed to
> > arch/powerpc.  Fine. The merge algorithms handled all the unchanged
> > files properly.  However arch/ppc64/Kconfig was modified and the
> > merging was left unresolved.  In fact there is no file no merge
> > against (because it moved).  So 'git-ls-files -u' only shows stage 1+3
> > (no stage 2, of course).
> >
> > How do I merge those changes?  I don't know enough about all the git
> > commands to figure this out, especially how to take advantage of the
> > stage 1, 2, and 3 files.
>
> I do not work on the kernel myself so I'll probably get the
> details wrong, but I am guessing here is what you have:
>
Sorry, I should have been more specific.  How about this:
1) 2.6.14-test branch modifies arch/ppc64/Kconfig
2) master branch renames arch/ppc64/Kconfig to arch/powerpc/Kconfig
3) 2.6.14-test and master branch share a common ancestor v2.6.14
4) Need to merge change arch/ppc64/Kconfig to arch/powerpc/Kconfig

$ git ls-files -u
100644 c658650af429672267409508b02b38754c11a40f 1       arch/ppc64/Kconfig
100644 8abf1118ebbd59954d098d87679114ffda0e75cb 3       arch/ppc64/Kconfig
***Notice no stage 2

$ ls -ld arch/p*
drwxrwxr-x   9 dzickus dzickus 4096 Dec 16 14:12 arch/parisc
drwxrwxr-x  11 dzickus dzickus 4096 Dec 16 14:26 arch/powerpc
drwxrwxr-x  15 dzickus dzickus 4096 Dec 16 14:26 arch/ppc
***Notice no ppc64 (was removed in master branch)

$ ls -ld arch/powerpc/Kconfig*
-rw-rw-r--  1 dzickus dzickus 24279 Dec 16 14:26 arch/powerpc/Kconfig
***Notice no backup or side copy of Kconfig

There were other conflicts with this merge (ones that weren't
automatically resolved) but those were easily merged using vi and
searching for <<<.  However this particular file is providing an
unusual case.

Let me know if this helps or not.

Cheers,
Don

Re: git merge questions

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> What puzzles me is that I think it is supposed to have done all
> the above for you.  Namely:
>
> 	$ ls -l arch/ppc64/Kconfig <1>
> 	$ git ls-files -s arch/ppc64/Kconfig <2>
>         $ ls -l arch/powerpc/Kconfig arch/powerpc/Kconfig~* <3>
>         $ git ls-files -s arch/powerpc/Kconfig <4>
>
> <1> should not remain --- recursive merge could have notied that
> the file was moved.
> <2> ditto.
> <3> arch/powerpc/Kconfig should be there with possibly merge
> conflict markers.
> <4> stage1 with merge base version (possibly renamed), stage2
> with test2 version, stage3 with your test version (definitely
> renamed).

I know what happened.  That Kconfig file was modified between
v2.6.14 and v2.6.15-rc4 beyond recognition, and the rename
detection did not think it was renamed.

The merge has this message:

    CONFLICT (delete/modify): arch/ppc64/Kconfig deleted in HEAD and
    modified in e14ee1ed34e25df6ea93b0bfb1bc4138cd26bea2. Version
    e14ee1ed34e25df6ea93b0bfb1bc4138cd26bea2 of arch/ppc64/Kconfig
    left in tree.

and then, the digging I suggested yields these:

$ ls -l arch/{ppc64,powerpc}/Kconfig
-rw-rw-r--  1 junio src 24279 Dec 16 12:55 arch/powerpc/Kconfig
-rw-rw-r--  1 junio src 11027 Dec 16 12:58 arch/ppc64/Kconfig
$ git ls-files -s arch/ppc64/Kconfig arch/powerpc/Kconfig
100644 bb2efdd566a9d590d64184b10b097e4b7ed17e95 0       arch/powerpc/Kconfig
100644 c658650af429672267409508b02b38754c11a40f 1       arch/ppc64/Kconfig
100644 8abf1118ebbd59954d098d87679114ffda0e75cb 3       arch/ppc64/Kconfig
$ ls arch/{powerpc,ppc64}/Kconfig~*
ls: arch/powerpc/Kconfig~*: No such file or directory
ls: arch/ppc64/Kconfig~*: No such file or directory
$ git diff --theirs arch/ppc64/Kconfig
* Unmerged path arch/ppc64/Kconfig
diff --git a/arch/ppc64/Kconfig b/arch/ppc64/Kconfig

The merge algorithm thought your branch (that is, test2 which is
v2.6.15-rc4) removed the old ppc64/Kconfig path, but the other
branch (test1 which has diff between v2.6.14 and v2.6.14.4) made
updates to that file.  For the other path, it simply thought
your branch added a new file arch/powerpc/Kconfig while the
other branch did not do anything to that path between v2.6.14
and v2.6.14.4, so it merged it already.

The result you want in this case is to merge changes between
c65865 (stage1 of old path) and 8abf11 (stage3 of old path) into
bb2efd (the latest contents of the new path) and register it as
the result of merge for arch/powerpc/Kconfig, and remove
arch/ppc64/Kconfig.  So the sequence would be:

$ orig=$(git unpack-file c65865)
$ from_test=$(git unpack-file 8abf11)
$ merge $from_test $orig arch/powerpc/Kconfig
merge: warning: conflicts during merge

After resolving the conflict in $from_test file, and other conflicts:

$ ed $from_test
$ cat $from_tset >arch/ppc64/Kconfig
$ rm arch/powerpc/Kconfig
$ rm $orig $from_test
$ git update-index --add --remove arch/ppc64/Kconfig arch/powerpc/Kconfig
# also mark paths you hand-resolved such as Makefile, drivers/pcmcia/i82365.c
# etc. with "git update-index" here.
$ git commit

would commit the result of the merge.

Re: git merge questions

[Don Zickus]

> and then, the digging I suggested yields these:
>
> $ ls -l arch/{ppc64,powerpc}/Kconfig
> -rw-rw-r--  1 junio src 24279 Dec 16 12:55 arch/powerpc/Kconfig
> -rw-rw-r--  1 junio src 11027 Dec 16 12:58 arch/ppc64/Kconfig
> $ git ls-files -s arch/ppc64/Kconfig arch/powerpc/Kconfig
> 100644 bb2efdd566a9d590d64184b10b097e4b7ed17e95 0       arch/powerpc/Kconfig
> 100644 c658650af429672267409508b02b38754c11a40f 1       arch/ppc64/Kconfig
> 100644 8abf1118ebbd59954d098d87679114ffda0e75cb 3       arch/ppc64/Kconfig
> $ ls arch/{powerpc,ppc64}/Kconfig~*
> ls: arch/powerpc/Kconfig~*: No such file or directory
> ls: arch/ppc64/Kconfig~*: No such file or directory
> $ git diff --theirs arch/ppc64/Kconfig
> * Unmerged path arch/ppc64/Kconfig
> diff --git a/arch/ppc64/Kconfig b/arch/ppc64/Kconfig
>
> The merge algorithm thought your branch (that is, test2 which is
> v2.6.15-rc4) removed the old ppc64/Kconfig path, but the other
> branch (test1 which has diff between v2.6.14 and v2.6.14.4) made
> updates to that file.  For the other path, it simply thought
> your branch added a new file arch/powerpc/Kconfig while the
> other branch did not do anything to that path between v2.6.14
> and v2.6.14.4, so it merged it already.
>
> The result you want in this case is to merge changes between
> c65865 (stage1 of old path) and 8abf11 (stage3 of old path) into
> bb2efd (the latest contents of the new path) and register it as
> the result of merge for arch/powerpc/Kconfig, and remove
> arch/ppc64/Kconfig.  So the sequence would be:
>
> $ orig=$(git unpack-file c65865)
> $ from_test=$(git unpack-file 8abf11)
> $ merge $from_test $orig arch/powerpc/Kconfig
> merge: warning: conflicts during merge
>
> After resolving the conflict in $from_test file, and other conflicts:
>
> $ ed $from_test
> $ cat $from_tset >arch/ppc64/Kconfig
> $ rm arch/powerpc/Kconfig
> $ rm $orig $from_test
> $ git update-index --add --remove arch/ppc64/Kconfig arch/powerpc/Kconfig
> # also mark paths you hand-resolved such as Makefile, drivers/pcmcia/i82365.c
> # etc. with "git update-index" here.
> $ git commit
>
> would commit the result of the merge.
>

Wow. That makes sense then.  All your digging techniques seem to be
fairly straightforward.   Now say I didn't know the name of the rename
and I had to dig that up.  Would the following be the right way or is
there something easier?

%git log <renamed file>  #grab the last commit id from here
%git-diff-tree -p <last commit id>  # search for the new file

Thanks again,
Don

Re: git merge questions

[Junio C Hamano]

Don Zickus <dzickus@gmail.com> writes:

>> and then, the digging I suggested yields these:
>>
>> $ ls -l arch/{ppc64,powerpc}/Kconfig
>> -rw-rw-r--  1 junio src 24279 Dec 16 12:55 arch/powerpc/Kconfig
>> -rw-rw-r--  1 junio src 11027 Dec 16 12:58 arch/ppc64/Kconfig
>> $ git ls-files -s arch/ppc64/Kconfig arch/powerpc/Kconfig
>> 100644 bb2efdd566a9d590d64184b10b097e4b7ed17e95 0       arch/powerpc/Kconfig
>> 100644 c658650af429672267409508b02b38754c11a40f 1       arch/ppc64/Kconfig
>> 100644 8abf1118ebbd59954d098d87679114ffda0e75cb 3       arch/ppc64/Kconfig
>> ..
>> The result you want in this case is to merge changes between
>> c65865 (stage1 of old path) and 8abf11 (stage3 of old path) into
>> bb2efd (the latest contents of the new path) and register it as
>> the result of merge for arch/powerpc/Kconfig, and remove
>> arch/ppc64/Kconfig.  So the sequence would be:
>>
>> $ orig=$(git unpack-file c65865)
>> $ from_test=$(git unpack-file 8abf11)
>> $ merge $from_test $orig arch/powerpc/Kconfig
>> merge: warning: conflicts during merge

I suspect there might be a room for improvement to make this
easier with a new command, if this becomes a common pattern.

Something like:

	$ git resolve-renamed-path arch/ppc64/Kconfig arch/powerpc/Kconfig

to mean "I want the result of this merge to rename ppc64/Kconfig
to powerpc/Kconfig", perhaps?  There are three cases: (1) we
renamed they didn't --- this is the case we are looking at and
there will be stage1 and stage3 but not stage2 for the old path,
and stage0 for the new path; (2) they renamed we didn't --- this
would happen if you pulled test2 into test, and there will be
stage1 and stage2 but not stage3 for the old path, and stage0
for the new path; (3) both of us renamed.

The third case is handled by the merge command automatically.
Old path will not remain to bother you even if the merge of the
new path needs hand-resolving, so you do not need
resolve-renamed-path command to deal with that case.

> Now say I didn't know the name of the rename and I had to dig
> that up.  Would the following be the right way or is there
> something easier?
>
> %git log <renamed file>  #grab the last commit id from here
> %git-diff-tree -p <last commit id>  # search for the new file

That would work.

Or an explicit rename detection to see where many of the
neighbouring paths moved (this is very expensive):

	mb=$(git merge-base test test2)
        git diff-tree -r --diff-filter=R -M -l0 --name-status $mb test2

Or the diff-tree between the merge base and your current tree
and perhaps the other tree (this may give a lot of cruft):

	mb=$(git merge-base test test2)
        git diff-tree -r --diff-filter=A --name-status $mb test2

Re: git merge questions

[Johannes Schindelin]

Hi,

just a thought: maybe in this case -- git fails to recognize a rename -- 
Pasky's idea would have some merit. You could then provide git with some 
extra information a la .git/info/grafts: "Even if you, git, do not believe 
it: this file *was* renamed from blah to blop".

Ciao,
Dscho

Re: git merge questions

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> just a thought: maybe in this case -- git fails to recognize a rename -- 
> Pasky's idea would have some merit. You could then provide git with some 
> extra information a la .git/info/grafts: "Even if you, git, do not believe 
> it: this file *was* renamed from blah to blop".

Yeah, except small details such as: where does it record it, and
how does the information presented to the user, and how does the
user tell git that information should be used?

What would be interesting would be to extend on this thing I
just wrote:

    Something like:

        $ git resolve-renamed-path arch/ppc64/ arch/powerpc/

    to mean "I want the result of this merge to rename ppc64/Kconfig
    to powerpc/Kconfig", perhaps?

I think this would work very nicely even without rename
detectino by the recursive strategy.  What would happen with
resolve strategy is that unchanged paths are removed from
arch/ppc64 and added to arch/powerpc by the usual read-tree
merge rules, and all paths (not just unrecognizable renames --
because resolve would not even try) that have been changed on
the "test" branch would be in stage1+stage3 state in arch/ppc64,
while the corresponding ones in arch/powerpc are collapsed
("only added in test2 branch") to stage0.  The fictional
resolve-renamed-path command (notice that I removed the explicit
"Kconfig" from the sample command line) could go through the
index file, looking for paths that arch/powerpc/ has stage0 and
arch/ppc64 has stage1+3, and perform the renaming merge at that
point.

Re: git merge questions

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> just a thought: maybe in this case -- git fails to recognize a rename -- 
> Pasky's idea would have some merit.

It is not as simple as that.  If you look at the output of the
following command, you would understand why.

	$ git rev-list ^$(git merge-base test test2) test2 -- \
          arch/powerpc/Kconfig arch/ppc64/Kconfig |
          git diff-tree --pretty -C -r --stdin --abbrev --name-status \
          arch/powerpc/Kconfig arch/ppc64/Kconfig

The transition happened over time with multiple commits.

First ppc64/Kconfig was somewhat stripped of its contents,
starting at this commit:

    diff-tree bcdd1ea... (from 5bfc826...)
    Author: Stephen Rothwell <sfr@canb.auug.org.au>
    Date:   Mon Sep 19 23:13:24 2005 +1000

        [PATCH] powerpc: Move arch/ppc*/oprofile/Kconfig to arch/powerpc

        These files are identical.

        Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
        Signed-off-by: Paul Mackerras <paulus@samba.org>

    M       arch/ppc64/Kconfig

and after a lot of hard work, powerpc/Kconfig gets
created.

    diff-tree 14cf11a... (from e5baa39...)
    Author: Paul Mackerras <paulus@samba.org>
    Date:   Mon Sep 26 16:04:21 2005 +1000

        powerpc: Merge enough to start building in arch/powerpc.

        This creates the directory structure under arch/powerpc and a bunch
        of Kconfig files...

    A       arch/powerpc/Kconfig

After that both continues to exist for some time, and finally
ppc64/Kconfig gets deleted with this one:

    diff-tree 7568cb4... (from c55377e...)
    Author: Paul Mackerras <paulus@samba.org>
    Date:   Mon Nov 14 17:30:17 2005 +1100

        powerpc: Move most remaining ppc64 files over to arch/powerpc

        Also deletes files in arch/ppc64 that are no longer used now that
        we don't compile with ARCH=ppc64 any more.

        Signed-off-by: Paul Mackerras <paulus@samba.org>

    M       arch/powerpc/Kconfig
    D       arch/ppc64/Kconfig


You cannot record "this is the rename" by attributing that
information to one particular commit.  Even looking at the
commit history one by one you cannot really say powerpc/Kconfig
is a rename of ppc64/Kconfig.  I suspect that it is not really a
rename --- from reading of the log messages, its original
contents were moved around and scattered over to other Kconfig
files in the tree, or along with other configuration pieces got
consolidated into powerpc/Kconfig, or perhaps both.