git-cherry-pick problem - directory not touched by commit is marked "added by us"

git-cherry-pick problem - directory not touched by commit is marked "added by us"

[Hakim Cassimally]

I got into a bit of trouble with a git-cherry-pick last night, and
though mugwump
and others on #git helped me as far as a workaround, I'm still very confused,
and would like to make sure it doesn't happen again

I'm fairly sure the problem is due to my ignorance, so I'll firstly describe
the mess I've made of my repo :-)

BACKGROUND
==========

 * I had a (trunk) branch that contained most of the code
 * I started to experiment with porting our .cgi code to a framework and later,
after the fact, created (experimental) off (trunk)
 * As experimental has of course taken longer than expected, I've since forked
(stable) again from (trunk), but from a commit before the experimental stages
started.

 * I've developed Feature "X", which took advantage of some of the experimental
refactoring... so of course I developed it in (experimental)...

 * I now want to port it back to (stable) so I can release it...

 * I was recommended to use 'git rebase -i' but didn't understand how

 * ... so I was also recommended to use 'git cherry-pick'.  As there were
only 6 relevant commits, I chose to do that

WHAT HAPPENS
============

When I'm in (stable), and try to cherry-pick the change from (experimental),
git thinks that I'm making a massive number of changes in a directory that
wasn't touched by the relevant commit.

Here's a sample transcript which hopefully makes sense: I've included output
of 'git show' and 'git whatchanged' which seem to suggest that the commit
I'm cherry-picking shouldn't be changing these files.

    (stable) $ git --version
    git version 1.6.6
    # I tried previously on 1.6.0.4 but upgraded in case it helped

    (stable) $ git status
    # On branch stable
    # nothing to commit (working directory clean)

    (stable) $ git show --stat 301afce1
    commit 301afce1c78380276d208077ef4ec76b469c1024
    Author: osfameron <...>
    Date:   Wed Dec 23 23:45:20 2009 +0000

        Proof of concept for import module (parse Excel)

     bin/upload_module.pl |  142
++++++++++++++++++++++++++++++++++++++++++++++++++
     1 files changed, 142 insertions(+), 0 deletions(-)

    (stable) $ git whatchanged -1 301afce1
    commit 301afce1c78380276d208077ef4ec76b469c1024
    Author: osfameron <...>
    Date:   Wed Dec 23 23:45:20 2009 +0000

        Proof of concept for import module (parse Excel)

    :000000 100644 0000000... c90e261... A  bin/upload_module.pl

    (stable) $ git cherry-pick 301afce1
    Finished one cherry-pick.
    www/client/css/admin-clean.css: needs merge
    www/client/css/admin.css: needs merge
    www/client/css/error.css: needs merge
    www/client/css/public.css: needs merge
    www/client/css/user-clean.css: needs merge
        <...snip>
    www/client/css/admin-clean.css: unmerged
(832c41e99d19f2dd39c9cf3709f14619ffab24b8)
    www/client/css/admin.css: unmerged
(8e7cd850bf40d1a921b1f62ce0945abd374fa55d)
    www/client/css/error.css: unmerged
(88945d05640c0820c9cf68922392573d4ac07b2c)
        <...snip>
    ...
    error: Error building trees

    (stable) $ git status
    # On branch stable
    # Changes to be committed:
    #   (use "git reset HEAD <file>..." to unstage)
    #
    #       new file:   bin/upload_module.pl
    #
    # Unmerged paths:
    #   (use "git reset HEAD <file>..." to unstage)
    #   (use "git add <file>..." to mark resolution)
    #
    #       added by us:        www/client/css/admin-clean.css
    #       added by us:        www/client/css/admin.css
    #       added by us:        www/client/css/error.css
    #       added by us:        www/client/css/public.css
        <...snip>

As mugwump suggested, I can resolve by doing

(stable) $ git checkout HEAD www/

And then proceed by commiting that cherry-pick.  This is how I've fixed it.
(Slightly annoying to repeat/lather/rinse for 6 separate commits, but works).

Oddly, though I've done some messing about in the www/ tree, these files exist
in both, for example "www/client/css/admin.css" exists in both (experimental)
and in (stable).

Please let me know if I should be submitting different or more detailed
information to describe the problem better!
Any help in working out what I'm doing wrong would be really appreciated,

osfameron

Re: git-cherry-pick problem - directory not touched by commit is marked "added by us"

[Sam Vilain]

On Tue, 2010-01-05 at 12:33 +0000, Hakim Cassimally wrote:
> I got into a bit of trouble with a git-cherry-pick last night, and
> though mugwump
> and others on #git helped me as far as a workaround, I'm still very confused,
> and would like to make sure it doesn't happen again

Yes, that was me.  I'm very confused by the bug, too.

 [...]
> WHAT HAPPENS
> ============
> 
> When I'm in (stable), and try to cherry-pick the change from (experimental),
> git thinks that I'm making a massive number of changes in a directory that
> wasn't touched by the relevant commit.
 [...]
>     (stable) $ git --version
>     git version 1.6.6
>     # I tried previously on 1.6.0.4 but upgraded in case it helped
> 
>     (stable) $ git status
>     # On branch stable
>     # nothing to commit (working directory clean)
> 
>     (stable) $ git show --stat 301afce1
>     commit 301afce1c78380276d208077ef4ec76b469c1024
>     Author: osfameron <...>
>     Date:   Wed Dec 23 23:45:20 2009 +0000
> 
>         Proof of concept for import module (parse Excel)
> 
>      bin/upload_module.pl |  142
> ++++++++++++++++++++++++++++++++++++++++++++++++++
>      1 files changed, 142 insertions(+), 0 deletions(-)
> 
>     (stable) $ git whatchanged -1 301afce1
>     commit 301afce1c78380276d208077ef4ec76b469c1024
>     Author: osfameron <...>
>     Date:   Wed Dec 23 23:45:20 2009 +0000
> 
>         Proof of concept for import module (parse Excel)
> 
>     :000000 100644 0000000... c90e261... A  bin/upload_module.pl

So by here, we know that the commit only affects that single file.  No
funny stuff like mode changes of other files.

>     (stable) $ git cherry-pick 301afce1
>     Finished one cherry-pick.
>     www/client/css/admin-clean.css: needs merge
>     www/client/css/admin.css: needs merge
>     www/client/css/error.css: needs merge
>     www/client/css/public.css: needs merge
>     www/client/css/user-clean.css: needs merge
>         <...snip>
>     www/client/css/admin-clean.css: unmerged
> (832c41e99d19f2dd39c9cf3709f14619ffab24b8)
>     www/client/css/admin.css: unmerged
> (8e7cd850bf40d1a921b1f62ce0945abd374fa55d)
>     www/client/css/error.css: unmerged
> (88945d05640c0820c9cf68922392573d4ac07b2c)
>         <...snip>
>     ...
>     error: Error building trees

Then, wham, these files want to be changed.  What is the diff-tree
inside revert/cherry-pick doing differently to the one in log?

>     (stable) $ git status
>     # On branch stable
>     # Changes to be committed:
>     #   (use "git reset HEAD <file>..." to unstage)
>     #
>     #       new file:   bin/upload_module.pl
>     #
>     # Unmerged paths:
>     #   (use "git reset HEAD <file>..." to unstage)
>     #   (use "git add <file>..." to mark resolution)
>     #
>     #       added by us:        www/client/css/admin-clean.css
>     #       added by us:        www/client/css/admin.css
>     #       added by us:        www/client/css/error.css
>     #       added by us:        www/client/css/public.css
>         <...snip>

Hakim, one more useful thing in this situation would be to show the
state of these files in the index;

 git ls-files -s -u

Also take the 'git ls-tree -r HEAD', 'git ls-tree -r 301afce1' and 'git
ls-tree -r 301afce1^' output, and grep for the files in question.  The
answer (or at least a bug triage) should be in the output of those
commands.

You can reproduce exactly the merge by making a new branch at the
position where you were attempting to land the cherry pick before, and
checking that branch out.

One last test, would be to check that it happens on a clean clone of the
repository.  I don't think that you're hitting any repository-local
behaviour (eg, the ability to mark certain files as "always ignore") but
you never know.  These commands are all being run on the same working
copy, right?

Thanks for your persistence,
Sam

Re: git-cherry-pick problem - directory not touched by commit is marked "added by us"

[Hakim Cassimally]

2010/1/5 Sam Vilain <sam@vilain.net>:
> On Tue, 2010-01-05 at 12:33 +0000, Hakim Cassimally wrote:
>> I got into a bit of trouble with a git-cherry-pick last night, and
>> though mugwump
>> and others on #git helped me as far as a workaround,
> Yes, that was me.  I'm very confused by the bug, too.
>
>  [...]
>> WHAT HAPPENS
>> ============
>>
>> When I'm in (stable), and try to cherry-pick the change from (experimental),
>> git thinks that I'm making a massive number of changes in a directory that
>> wasn't touched by the relevant commit.
>  [...]
>>     (stable) $ git --version
>>     git version 1.6.6
>>     # I tried previously on 1.6.0.4 but upgraded in case it helped
>>
>>     (stable) $ git status
>>     # On branch stable
>>     # nothing to commit (working directory clean)
>>
>>     (stable) $ git show --stat 301afce1
>>     commit 301afce1c78380276d208077ef4ec76b469c1024
>>     Author: osfameron <...>
>>     Date:   Wed Dec 23 23:45:20 2009 +0000
>>
>>         Proof of concept for import module (parse Excel)
>>
>>      bin/upload_module.pl |  142
>> ++++++++++++++++++++++++++++++++++++++++++++++++++
>>      1 files changed, 142 insertions(+), 0 deletions(-)
>>
>>     (stable) $ git whatchanged -1 301afce1
>>     commit 301afce1c78380276d208077ef4ec76b469c1024
>>     Author: osfameron <...>
>>     Date:   Wed Dec 23 23:45:20 2009 +0000
>>
>>         Proof of concept for import module (parse Excel)
>>
>>     :000000 100644 0000000... c90e261... A  bin/upload_module.pl
>
> So by here, we know that the commit only affects that single file.  No
> funny stuff like mode changes of other files.
>
>>     (stable) $ git cherry-pick 301afce1
>>     Finished one cherry-pick.
>>     www/client/css/admin.css: needs merge
>>         <...snip>
>>     www/client/css/admin.css: unmerged
>> (8e7cd850bf40d1a921b1f62ce0945abd374fa55d)
>>         <...snip>
>>     ...
>>     error: Error building trees
>
> Then, wham, these files want to be changed.  What is the diff-tree
> inside revert/cherry-pick doing differently to the one in log?

> Hakim, one more useful thing in this situation would be to show the
> state of these files in the index;
>
>  git ls-files -s -u

 $ git ls-files -s -u www/client/css/admin.css # only staged/unmerged
  100755 8e7cd850bf40d1a921b1f62ce0945abd374fa55d 2
www/client/css/admin.css

(i.e. when run after the failed cherry-pick)

> Also take the 'git ls-tree -r HEAD', 'git ls-tree -r 301afce1' and 'git
> ls-tree -r 301afce1^' output, and grep for the files in question.  The
> answer (or at least a bug triage) should be in the output of those
> commands.

  $ git ls-tree HEAD | grep www/client/css/admin.css
  100755 blob 8e7cd850bf40d1a921b1f62ce0945abd374fa55d
www/client/css/admin.css

There's no entry in git ls-tree 301afce1 or 301afce1^1 though.  I'd
guess that's significant, but I don't know what answer it suggests...

(However the file exists in both (stable) and (experimental)...
and is identical in both).

> You can reproduce exactly the merge by making a new branch at the
> position where you were attempting to land the cherry pick before, and
> checking that branch out.

I actually did

  $ cd ..
  $ git clone repo/ repo_backup_4Jan

beforehand, so this should also be an exact reproduction, I think?

> One last test, would be to check that it happens on a clean clone of the
> repository.  I don't think that you're hitting any repository-local
> behaviour (eg, the ability to mark certain files as "always ignore") but
> you never know.  These commands are all being run on the same working
> copy, right?

And see above, should be an entirely clean clone.

Thanks for the suggestions!

Hakim / osfameron

Re: git-cherry-pick problem - directory not touched by commit is marked "added by us"

[Junio C Hamano]

Hakim Cassimally <hakim.cassimally@gmail.com> writes:

>>>     (stable) $ git cherry-pick 301afce1
>>>     Finished one cherry-pick.
>>>     www/client/css/admin.css: needs merge
>>>         <...snip>
>>>     www/client/css/admin.css: unmerged (8e7cd850bf40d1a921b1f62ce0945abd374fa55d)
>>>         <...snip>
>>>     ...
>>>     error: Error building trees
>>
>  $ git ls-files -s -u www/client/css/admin.css # only staged/unmerged
>   100755 8e7cd850bf40d1a921b1f62ce0945abd374fa55d 2
> www/client/css/admin.css
>
> (i.e. when run after the failed cherry-pick)
>
>> Also take the 'git ls-tree -r HEAD', 'git ls-tree -r 301afce1' and 'git
>> ls-tree -r 301afce1^' output, and grep for the files in question.  The
>> answer (or at least a bug triage) should be in the output of those
>> commands.
>
>   $ git ls-tree HEAD | grep www/client/css/admin.css
>   100755 blob 8e7cd850bf40d1a921b1f62ce0945abd374fa55d
> www/client/css/admin.css
>
> There's no entry in git ls-tree 301afce1 or 301afce1^1 though.  I'd
> guess that's significant, but I don't know what answer it suggests...
>
> (However the file exists in both (stable) and (experimental)...
> and is identical in both).
> ....

So in short:

 * commit 301a added a new path bin/upload_module.pl;

 * the state you cherry-picked this commit to was clean (i.e. before
   running cherry-pick, "git status" reported no local changes to the
   index nor to the work tree);

 * the commit, the index, and the work tree before running this
   cherry-pick had www/client/css/admin.css file, which also existed in
   301a and 301a^, but all of these three commits had the same contents.

A simple minded attempt to reproduce this (attached below) by preparing a
commit that adds one new file and attempting to transplant to an unrelated
commit that doesn't have the file didn't work; I have been scratching my
head.

What "cherry-pick" internally does is to run:

    git merge-recursive 301a^ -- HEAD 301a

That is, "We are at HEAD; consolidate the change since 301a^ to 301a into
our state, and leave the result in the index and the work tree".  Then it
commits the result.  One thing to try is to see if this gives the same
kind of breakage.

The only other thing that _may_ be involved is if there are paths that
turned between directories and files on the two histories, or perhaps
there are paths like "www-frotz", "www.frotz", etc (i.e. "www" followed by
a byte whose ASCII value comes before '/') involved, and unpack_trees()
machinery the merge-recursive internally uses are getting confused about
D/F conflicts.  To diagnose it, it would be helpful to get the full output
from these two commands:

    git ls-tree -r -t HEAD (before cherry-pick)
    git ls-tree -r -t 301a (the commit you are transplanting)

Thanks

---

diff --git a/t/t3506-cherry-pick-addremove.sh b/t/t3506-cherry-pick-addremove.sh
new file mode 100755
index 0000000..e7dcd77
--- /dev/null
+++ b/t/t3506-cherry-pick-addremove.sh
@@ -0,0 +1,26 @@
+#!/bin/sh
+
+test_description='unrelated files added by our side'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	test_commit A &&
+	git branch side &&
+	test_commit B &&
+	test_commit C &&
+
+	git checkout side &&
+	test_commit D &&
+	test_commit E &&
+
+	git checkout master
+'
+
+test_expect_success 'cherry-pick' '
+	git reset --hard C &&
+	git cherry-pick side &&
+	git grep E E.t
+'
+
+test_done

Re: git-cherry-pick problem - directory not touched by commit is marked "added by us"

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> What "cherry-pick" internally does is to run:
>
>     git merge-recursive 301a^ -- HEAD 301a
>
> That is, "We are at HEAD; consolidate the change since 301a^ to 301a into
> our state, and leave the result in the index and the work tree".  Then it
> commits the result.  One thing to try is to see if this gives the same
> kind of breakage.

There actually is another possibility; we used to run inside "cherry-pick"

    git merge-resolve 301a^ -- HEAD 301a

instead.  The request is the same but it uses a different algorithm, so if
one fails and one succeeds, that might give us a better clue to figure out
what is going on.

Re: git-cherry-pick problem - directory not touched by commit is marked "added by us"

[Hakim Cassimally]

(woops, sent just to Junio, resending to list)

Hi,

Thanks for the suggestions, I've tried both those commands, and they
break too :-(

2010/1/7 Junio C Hamano <gitster@pobox.com>:
> Junio C Hamano <gitster@pobox.com> writes:
>> What "cherry-pick" internally does is to run:
>>
>>     git merge-recursive 301a^ -- HEAD 301a
>>
>> That is, "We are at HEAD; consolidate the change since 301a^ to 301a into
>> our state, and leave the result in the index and the work tree".  Then it
>> commits the result.  One thing to try is to see if this gives the same
>> kind of breakage.

    $ git merge-recursive 301a^ -- HEAD 301a
    # woo! no response!  promising
    $ git status
# On branch cgi_branch
<snip>
#       new file:   bin/upload_module.pl
#
# Unmerged paths:
#       added by us:        www/client/css/admin-clean.css
#       added by us:        www/client/css/admin.css
<snip dozens of spurious "added by us">

> There actually is another possibility; we used to run inside "cherry-pick"
>
>    git merge-resolve 301a^ -- HEAD 301a
>
> instead.  The request is the same but it uses a different algorithm, so if
> one fails and one succeeds, that might give us a better clue to figure out
> what is going on.

  $ git merge-resolve 301a^ -- HEAD 301a
  Trying simple merge.
  Simple merge failed, trying Automatic merge.
  error: www/client: is a directory - add individual files instead
  fatal: Unable to process path www/client
  error: www/css: is a directory - add individual files instead
  fatal: Unable to process path www/css
  error: www/images: is a directory - add individual files instead
  fatal: Unable to process path www/images
  error: www/js: is a directory - add individual files instead
  fatal: Unable to process path www/js
  fatal: merge program failed

At least the latter fails more informatively?

Thanks again for the help and suggestions ;-)

osfameron