If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

[Dakota Hawkins]

Below is a simple reproduction of the issue.

The _real_ problem is that this is how our pull request merges work,
they're not allowed to do fast-forward merges. To work around this we
are having to split this up into two pull requests/merges: One that
copies the submodules to the new location and includes any fixes
required to support the move, and a second that removes the old
locations.

## Setup steps
git clone https://github.com/dakotahawkins/submodule-move-merge-bug-main-repo.git
cd submodule-move-merge-bug-main-repo
    ## How it was initially constructed
    # git submodule add ../submodule-move-merge-bug-submodule-repo.git
./submodule-location-1
    # git commit -m "Added submodule in its initial location"
    # git push
    # git checkout -b move-submodule
    # git mv ./submodule-location-1 ./submodule-location-2
    # git commit -m "Moved submodule"
    # git push --set-upstream origin move-submodule
git branch move-submodule origin/move-submodule

## Test fast-forward merge, this will work
git checkout -b merge-ff-test master # warning: unable to rmdir
submodule-location-2: Directory not empty
rm -rf ./submodule-location-2
git merge --ff-only move-submodule

## Test no-fast-forward merge, this will fail with conflicts:
git checkout -b merge-no-ff-test master
git merge --no-ff move-submodule
    # Auto-merging submodule-location-2
    # Adding as submodule-location-2~move-submodule instead
    # Automatic merge failed; fix conflicts and then commit the result.
git status
    # On branch merge-no-ff-test
    # You have unmerged paths.
    #   (fix conflicts and run "git commit")
    #   (use "git merge --abort" to abort the merge)
    #
    # Changes to be committed:
    #
    #         modified:   .gitmodules
    #         deleted:    submodule-location-1
    #
    # Unmerged paths:
    #   (use "git add <file>..." to mark resolution)
    #
    #         added by us:     submodule-location-2
    #
    # fatal: Not a git repository: 'submodule-location-1/.git'
    # Submodule changes to be committed:
    #
    # * submodule-location-1 07fec24...0000000:

Re: If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

[Dakota Hawkins]

Is there any additional information I could provide that would be helpful?

Dakota

On Fri, Sep 2, 2016 at 3:22 PM, Dakota Hawkins <dakotahawkins@gmail.com> wrote:
> Below is a simple reproduction of the issue.
>
> The _real_ problem is that this is how our pull request merges work,
> they're not allowed to do fast-forward merges. To work around this we
> are having to split this up into two pull requests/merges: One that
> copies the submodules to the new location and includes any fixes
> required to support the move, and a second that removes the old
> locations.
>
> ## Setup steps
> git clone https://github.com/dakotahawkins/submodule-move-merge-bug-main-repo.git
> cd submodule-move-merge-bug-main-repo
>     ## How it was initially constructed
>     # git submodule add ../submodule-move-merge-bug-submodule-repo.git
> ./submodule-location-1
>     # git commit -m "Added submodule in its initial location"
>     # git push
>     # git checkout -b move-submodule
>     # git mv ./submodule-location-1 ./submodule-location-2
>     # git commit -m "Moved submodule"
>     # git push --set-upstream origin move-submodule
> git branch move-submodule origin/move-submodule
>
> ## Test fast-forward merge, this will work
> git checkout -b merge-ff-test master # warning: unable to rmdir
> submodule-location-2: Directory not empty
> rm -rf ./submodule-location-2
> git merge --ff-only move-submodule
>
> ## Test no-fast-forward merge, this will fail with conflicts:
> git checkout -b merge-no-ff-test master
> git merge --no-ff move-submodule
>     # Auto-merging submodule-location-2
>     # Adding as submodule-location-2~move-submodule instead
>     # Automatic merge failed; fix conflicts and then commit the result.
> git status
>     # On branch merge-no-ff-test
>     # You have unmerged paths.
>     #   (fix conflicts and run "git commit")
>     #   (use "git merge --abort" to abort the merge)
>     #
>     # Changes to be committed:
>     #
>     #         modified:   .gitmodules
>     #         deleted:    submodule-location-1
>     #
>     # Unmerged paths:
>     #   (use "git add <file>..." to mark resolution)
>     #
>     #         added by us:     submodule-location-2
>     #
>     # fatal: Not a git repository: 'submodule-location-1/.git'
>     # Submodule changes to be committed:
>     #
>     # * submodule-location-1 07fec24...0000000:

Re: If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

[Stefan Beller]

On Fri, Sep 2, 2016 at 12:22 PM, Dakota Hawkins <dakotahawkins@gmail.com> wrote:
> Below is a simple reproduction of the issue.
>
> The _real_ problem is that this is how our pull request merges work,

So your workflow is the problem or is the actual bug just exposed in
your workflow?

> they're not allowed to do fast-forward merges. To work around this we
> are having to split this up into two pull requests/merges: One that
> copies the submodules to the new location and includes any fixes
> required to support the move, and a second that removes the old
> locations.
>
> ## Setup steps
> git clone https://github.com/dakotahawkins/submodule-move-merge-bug-main-repo.git
> cd submodule-move-merge-bug-main-repo
>     ## How it was initially constructed
>     # git submodule add ../submodule-move-merge-bug-submodule-repo.git
> ./submodule-location-1
>     # git commit -m "Added submodule in its initial location"
>     # git push
>     # git checkout -b move-submodule
>     # git mv ./submodule-location-1 ./submodule-location-2
>     # git commit -m "Moved submodule"
>     # git push --set-upstream origin move-submodule
> git branch move-submodule origin/move-submodule
>
> ## Test fast-forward merge, this will work
> git checkout -b merge-ff-test master # warning: unable to rmdir
> submodule-location-2: Directory not empty
> rm -rf ./submodule-location-2
> git merge --ff-only move-submodule
>

And no reset/rm in between, i.e. we still have
submodule-location-2 from  merge-ff-test still around?

> ## Test no-fast-forward merge, this will fail with conflicts:
> git checkout -b merge-no-ff-test master
> git merge --no-ff move-submodule
>     # Auto-merging submodule-location-2
>     # Adding as submodule-location-2~move-submodule instead
>     # Automatic merge failed; fix conflicts and then commit the result.
> git status
>     # On branch merge-no-ff-test
>     # You have unmerged paths.
>     #   (fix conflicts and run "git commit")
>     #   (use "git merge --abort" to abort the merge)
>     #
>     # Changes to be committed:
>     #
>     #         modified:   .gitmodules
>     #         deleted:    submodule-location-1
>     #
>     # Unmerged paths:
>     #   (use "git add <file>..." to mark resolution)
>     #
>     #         added by us:     submodule-location-2
>     #
>     # fatal: Not a git repository: 'submodule-location-1/.git'
>     # Submodule changes to be committed:
>     #
>     # * submodule-location-1 07fec24...0000000:

Re: If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

[Dakota Hawkins]

On Tue, Sep 6, 2016 at 3:00 PM, Stefan Beller <sbeller@google.com> wrote:
> On Fri, Sep 2, 2016 at 12:22 PM, Dakota Hawkins <dakotahawkins@gmail.com> wrote:
>> Below is a simple reproduction of the issue.
>>
>> The _real_ problem is that this is how our pull request merges work,
>
> So your workflow is the problem or is the actual bug just exposed in
> your workflow?

I believe the workflow just exposes the problem, which is that
generically for this case a fast-forward merge works without conflicts
while a non-fast-forward merge fails with conflicts. Sorry if this
confuses the issue, it's just how we experienced it so I wanted to add
that background.

>> ## Test fast-forward merge, this will work
>> git checkout -b merge-ff-test master # warning: unable to rmdir
>> submodule-location-2: Directory not empty
>> rm -rf ./submodule-location-2
>> git merge --ff-only move-submodule
>>
>
> And no reset/rm in between, i.e. we still have
> submodule-location-2 from  merge-ff-test still around?

That is true in that example, but somewhat immaterial. The first merge
was only to demonstrate that a fast-forward merge works without
conflict. The simplest reproduction is to skip that and get straight
to the failure case:

## Clone and setup branches
git clone https://github.com/dakotahawkins/submodule-move-merge-bug-main-repo.git
cd submodule-move-merge-bug-main-repo
git branch move-submodule origin/move-submodule
git checkout -b merge-no-ff-test master

## This will fail
git merge --no-ff move-submodule
# Auto-merging submodule-location-2
# Adding as submodule-location-2~move-submodule instead
# Automatic merge failed; fix conflicts and then commit the result.

Does that make things a little bit clearer?

Dakota

Re: If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

[Dakota Hawkins]

Any ideas on this anywhere?

In my opinion if a merge can fast-forward without conflict it should
trivially be able to non-fast-forward without conflict.

Also, I'm not too familiar/comfortable with mailing list etiquette,
and I don't want to be a bother by continuing to ping this thread.

Thanks,

Dakota

On Tue, Sep 6, 2016 at 4:02 PM, Dakota Hawkins <dakotahawkins@gmail.com> wrote:
> On Tue, Sep 6, 2016 at 3:00 PM, Stefan Beller <sbeller@google.com> wrote:
>> On Fri, Sep 2, 2016 at 12:22 PM, Dakota Hawkins <dakotahawkins@gmail.com> wrote:
>>> Below is a simple reproduction of the issue.
>>>
>>> The _real_ problem is that this is how our pull request merges work,
>>
>> So your workflow is the problem or is the actual bug just exposed in
>> your workflow?
>
> I believe the workflow just exposes the problem, which is that
> generically for this case a fast-forward merge works without conflicts
> while a non-fast-forward merge fails with conflicts. Sorry if this
> confuses the issue, it's just how we experienced it so I wanted to add
> that background.
>
>>> ## Test fast-forward merge, this will work
>>> git checkout -b merge-ff-test master # warning: unable to rmdir
>>> submodule-location-2: Directory not empty
>>> rm -rf ./submodule-location-2
>>> git merge --ff-only move-submodule
>>>
>>
>> And no reset/rm in between, i.e. we still have
>> submodule-location-2 from  merge-ff-test still around?
>
> That is true in that example, but somewhat immaterial. The first merge
> was only to demonstrate that a fast-forward merge works without
> conflict. The simplest reproduction is to skip that and get straight
> to the failure case:
>
> ## Clone and setup branches
> git clone https://github.com/dakotahawkins/submodule-move-merge-bug-main-repo.git
> cd submodule-move-merge-bug-main-repo
> git branch move-submodule origin/move-submodule
> git checkout -b merge-no-ff-test master
>
> ## This will fail
> git merge --no-ff move-submodule
> # Auto-merging submodule-location-2
> # Adding as submodule-location-2~move-submodule instead
> # Automatic merge failed; fix conflicts and then commit the result.
>
> Does that make things a little bit clearer?
>
> Dakota

Re: If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

[Stefan Beller]

On Fri, Sep 9, 2016 at 4:37 PM, Dakota Hawkins <dakotahawkins@gmail.com> wrote:
> Any ideas on this anywhere?
>
> In my opinion if a merge can fast-forward without conflict it should
> trivially be able to non-fast-forward without conflict.

Yes, I agree. Though the submodules still have a lot of sharp edges
that you can injure yourself with.

I am sorry to let you down here, but I did not have time and mental
capacity to actually dive into the issue further this week.
I hoped someone else would have picked up this thread and have
a good idea.

As you know currently the checkout doesn't touch submodules, i.e.
you have to run "git submodule update" whenever the submodule
changes. So when you checkout a different part of history, that moved
a submodule, this will fail as the submodule still resides at the old
place (and may have path name conflict with another thing)
and is not there at the new place.

I plan to fix that in the near future (read: teach git checkout to know
about submodules), mind that there is already a patch series for
exactly that out there in one of the branches of
https://github.com/jlehmann/git-submod-enhancements

This seems to be not an easy fix; Someone tried already and getting
it polished enough to include it upstream was not successful.

>
> Also, I'm not too familiar/comfortable with mailing list etiquette,
> and I don't want to be a bother by continuing to ping this thread.

Pinging is fine, as it is rather easy to ignore mails on a mailing list. ;)
I just don't know if it increases likelihood of someone responding.

Thanks,
Stefan

Re: If a branch moves a submodule, "merge --ff[-only]" succeeds while "merge --no-ff" fails with conflicts

[Dakota Hawkins]

> As you know currently the checkout doesn't touch submodules, i.e.
> you have to run "git submodule update" whenever the submodule
> changes. So when you checkout a different part of history, that moved
> a submodule, this will fail as the submodule still resides at the old
> place (and may have path name conflict with another thing)
> and is not there at the new place.

I _think_ (may be wrong) that you can reproduce this without ever
updating the submodules at all. I may be missing/forgetting something
in my config: I normally have global post-checkout and post-merge
hooks to run a submodule update, however I realized that and (I think)
disabled them for the second more concise reproduction I sent. I moved
them to a "no" subdirectory in my hook directory -- git doesn't
recursively look for hooks, does it? Anyway, I don't think that's a
difference in my case. It may be, I agree, if you happen to be
reproducing this issue locally in your working repository for some
reason.

That thought process makes sense to me because as I mentioned
initially these failures presented on our remote (bitbucket server, if
it matters) for a pull-request merge. I don't know for certain but I
don't believe the server's copy of the repo would need to update the
submodules at all, ever.

>>
>> Also, I'm not too familiar/comfortable with mailing list etiquette,
>> and I don't want to be a bother by continuing to ping this thread.
>
> Pinging is fine, as it is rather easy to ignore mails on a mailing list. ;)
> I just don't know if it increases likelihood of someone responding.

Apparently it got you on the hook! Thanks for taking the bait :)

Dakota