Interactive rebase with submodules

Interactive rebase with submodules

[John Keeping]

I've encountered a scenario where git rebase --interactive drops a 
commit which contains a modification to a submodule but no other changes.

This occurs when there is a conflict when applying the commit (for 
example if the submodule's history has been rewritten and you are 
rewriting the parent repository to match the new version of the submodule).

To clarify:

git rebase -i
# Edit a commit, switching submodule to an unrelated commit
git rebase --continue
# Conflict in submodule, checkout the correct submodule commit
git add path/to/submodule
# Only change in index is updated submodule
git rebase --continue
# No commit is created for the submodule change


This appears to be because the git-rebase--interactive script inspects 
whether there is anything to commit when `rebase --continue` is invoked 
by running:

     git diff-index --cached --quiet --ignore-submodules HEAD --

Is there a reason for the `--ignore-submodules` in this command? 
Removing that option results in the expected behaviour.


I can understand not updating submodules while running the rebase, but I 
expected that having resolved a conflict and added my change to the 
index it would be applied by `git rebase --continue`, as indeed it is if 
there happen to be other (non-submodule) changes in the same commit.


-- 
John

Re: Interactive rebase with submodules

[Jens Lehmann]

Am 17.01.2012 19:47, schrieb John Keeping:
> I've encountered a scenario where git rebase --interactive drops a commit which contains a modification to a submodule but no other changes.
> 
> This occurs when there is a conflict when applying the commit (for example if the submodule's history has been rewritten and you are rewriting the parent repository to match the new version of the submodule).
> 
> To clarify:
> 
> git rebase -i
> # Edit a commit, switching submodule to an unrelated commit
> git rebase --continue
> # Conflict in submodule, checkout the correct submodule commit
> git add path/to/submodule
> # Only change in index is updated submodule
> git rebase --continue
> # No commit is created for the submodule change
> 
> 
> This appears to be because the git-rebase--interactive script inspects whether there is anything to commit when `rebase --continue` is invoked by running:
> 
>     git diff-index --cached --quiet --ignore-submodules HEAD --

Thanks for pinning that down.

> Is there a reason for the `--ignore-submodules` in this command? Removing that option results in the expected behaviour.

Yes, removing it will help your use case but break others. The reason
for that is that because submodules are not updated during a rebase
it doesn't make sense to compare their HEAD to what is recorded in
the superproject, as that might have been changed by an earlier
commit. And as the submodules HEAD hasn't been updated back then,
it is stale and will always show up as modified (even if it wasn't).

> I can understand not updating submodules while running the rebase, but I expected that having resolved a conflict and added my change to the index it would be applied by `git rebase --continue`, as indeed it is if there happen to be other (non-submodule) changes in the same commit.

The irony is that you would have to update submodules (or at least
their HEAD and use "--ignore-submodules=dirty") while running rebase
to make that work in all cases ;-)

But just updating the HEAD would be dangerous as you would have to be
very careful to restore the submodules HEAD after the rebase, or the
submodule's work tree will be out of sync.

I suspect in the long run a rebase should, e.g. when invoked with
--recurse-submodules, update the submodules too and won't use the
--ignore-submodule option for diff anymore ... then everything
should Just Work. But until that happens, I don't see a solution
for your problem.

Re: Interactive rebase with submodules

[John Keeping]

On 17/01/12 21:29, Jens Lehmann wrote:
> Am 17.01.2012 19:47, schrieb John Keeping:
>> This appears to be because the git-rebase--interactive script inspects whether there is anything to commit when `rebase --continue` is invoked by running:
>>
>>      git diff-index --cached --quiet --ignore-submodules HEAD --
 >
>> Is there a reason for the `--ignore-submodules` in this command? Removing that option results in the expected behaviour.
>
> Yes, removing it will help your use case but break others. The reason
> for that is that because submodules are not updated during a rebase
> it doesn't make sense to compare their HEAD to what is recorded in
> the superproject, as that might have been changed by an earlier
> commit. And as the submodules HEAD hasn't been updated back then,
> it is stale and will always show up as modified (even if it wasn't).

Is this worse than the current behaviour?  If I perform a rebase where 
there is a (non-submodule) conflict in a commit where a submodule 
changes I can see something like:

# Changes to be committed:
#     modified:    path/to/submodule
#
# Unmerged paths:
#     both modified:      path/to/file
#
# Changes not staged for commit:
#     modified:    path/to/submodule (new commits)

This occurs if a later commit in the rebase will modify the submodule. 
In this case, `rebase --continue` correctly recreates the commit once I 
have resolved the conflict in the file, ignoring the unstaged submodule 
changes.

>> I can understand not updating submodules while running the rebase, but I expected that having resolved a conflict and added my change to the index it would be applied by `git rebase --continue`, as indeed it is if there happen to be other (non-submodule) changes in the same commit.
>
> The irony is that you would have to update submodules (or at least
> their HEAD and use "--ignore-submodules=dirty") while running rebase
> to make that work in all cases ;-)

I don't this this is the case, since diff-tree is being invoked with 
--cached won't it ignore changes in the work tree anyway?

> But just updating the HEAD would be dangerous as you would have to be
> very careful to restore the submodules HEAD after the rebase, or the
> submodule's work tree will be out of sync.

Just updating HEAD in the submodule without touching its work tree 
doesn't seem like a good idea.  I think it will cause a lot more 
confusion when running `git status` which will show unexpected modified 
content for the submodule.

Since I did not expect rebase to perform a submodule update, I was not 
surprised to see unstaged submodule changes when rebasing, but I did 
expect rebase to commit anything I had added to the index.


-- 
John

Re: Interactive rebase with submodules

[Jens Lehmann]

Am 18.01.2012 12:21, schrieb John Keeping:
> On 17/01/12 21:29, Jens Lehmann wrote:
>> Am 17.01.2012 19:47, schrieb John Keeping:
>>> This appears to be because the git-rebase--interactive script inspects whether there is anything to commit when `rebase --continue` is invoked by running:
>>>
>>>      git diff-index --cached --quiet --ignore-submodules HEAD --
>>
>>> Is there a reason for the `--ignore-submodules` in this command? Removing that option results in the expected behaviour.
>>
>> Yes, removing it will help your use case but break others. The reason
>> for that is that because submodules are not updated during a rebase
>> it doesn't make sense to compare their HEAD to what is recorded in
>> the superproject, as that might have been changed by an earlier
>> commit. And as the submodules HEAD hasn't been updated back then,
>> it is stale and will always show up as modified (even if it wasn't).
> 
> Is this worse than the current behaviour?

If we break established behavior that would be worse unless we have
a *very* good reason to do so. But please see below ...

>  If I perform a rebase where there is a (non-submodule) conflict in a commit where a submodule changes I can see something like:
> 
> # Changes to be committed:
> #     modified:    path/to/submodule
> #
> # Unmerged paths:
> #     both modified:      path/to/file
> #
> # Changes not staged for commit:
> #     modified:    path/to/submodule (new commits)
> 
> This occurs if a later commit in the rebase will modify the submodule. In this case, `rebase --continue` correctly recreates the commit once I have resolved the conflict in the file, ignoring the unstaged submodule changes.

Yeah, as rebase doesn't touch the submodules, that is expected.

>>> I can understand not updating submodules while running the rebase, but I expected that having resolved a conflict and added my change to the index it would be applied by `git rebase --continue`, as indeed it is if there happen to be other (non-submodule) changes in the same commit.
>>
>> The irony is that you would have to update submodules (or at least
>> their HEAD and use "--ignore-submodules=dirty") while running rebase
>> to make that work in all cases ;-)
> 
> I don't this this is the case, since diff-tree is being invoked with --cached won't it ignore changes in the work tree anyway?

Right, thanks for nudging me with the clue bat ...

I missed the "--cached" option and did not question if the code does
what the commit message of 6848d58c6 (where the --ignore-submodules
option was introduced) said:

    Ignore dirty submodule states during rebase and stash

    When rebasing or stashing, chances are that you do not care about
    dirty submodules, since they are not updated by those actions anyway.
    So ignore the submodules' states.

    Note: the submodule states -- as committed in the superproject --
    will still be stashed and rebased, it is _just_ the state of the
    submodule in the working tree which is ignored.

I think this logic misses the case when only submodule changes are left
in a commit.

>> But just updating the HEAD would be dangerous as you would have to be
>> very careful to restore the submodules HEAD after the rebase, or the
>> submodule's work tree will be out of sync.
> 
> Just updating HEAD in the submodule without touching its work tree doesn't seem like a good idea.  I think it will cause a lot more confusion when running `git status` which will show unexpected modified content for the submodule.

Yes, we agree here.

> Since I did not expect rebase to perform a submodule update, I was not surprised to see unstaged submodule changes when rebasing, but I did expect rebase to commit anything I had added to the index.

Right.

I'll have to add some tests for that case, but I doubt I'll manage that
today. Until I can provide a complete patch, this diff should fix your
problem (no, I did not test if that change is enough to fix the problem,
but at least it does not break the test suite ;-):

---------------8<--------------
diff --git a/git-rebase--interactive.sh b/git-rebase--interactive.sh
index 5812222..4546749 100644
--- a/git-rebase--interactive.sh
+++ b/git-rebase--interactive.sh
@@ -672,7 +672,7 @@ rearrange_squash () {
 case "$action" in
 continue)
        # do we have anything to commit?
-       if git diff-index --cached --quiet --ignore-submodules HEAD --
+       if git diff-index --cached --quiet HEAD --
        then
                : Nothing to commit -- skip this
        else

Re: Interactive rebase with submodules

[John Keeping]

On 18/01/12 20:38, Jens Lehmann wrote:
> Am 18.01.2012 12:21, schrieb John Keeping:
>> On 17/01/12 21:29, Jens Lehmann wrote:
>>> Am 17.01.2012 19:47, schrieb John Keeping:
>>>> I can understand not updating submodules while running the rebase, but I expected that having resolved a conflict and added my change to the index it would be applied by `git rebase --continue`, as indeed it is if there happen to be other (non-submodule) changes in the same commit.
>>>
>>> The irony is that you would have to update submodules (or at least
>>> their HEAD and use "--ignore-submodules=dirty") while running rebase
>>> to make that work in all cases ;-)
>>
>> I don't this this is the case, since diff-tree is being invoked with --cached won't it ignore changes in the work tree anyway?
>
> Right, thanks for nudging me with the clue bat ...
>
> I missed the "--cached" option and did not question if the code does
> what the commit message of 6848d58c6 (where the --ignore-submodules
> option was introduced) said:
>
>      Ignore dirty submodule states during rebase and stash
>
>      When rebasing or stashing, chances are that you do not care about
>      dirty submodules, since they are not updated by those actions anyway.
>      So ignore the submodules' states.
>
>      Note: the submodule states -- as committed in the superproject --
>      will still be stashed and rebased, it is _just_ the state of the
>      submodule in the working tree which is ignored.
>
> I think this logic misses the case when only submodule changes are left
> in a commit.

Yes, this is precisely the case that doesn't work as I expect.

>>> But just updating the HEAD would be dangerous as you would have to be
>>> very careful to restore the submodules HEAD after the rebase, or the
>>> submodule's work tree will be out of sync.
>>
>> Just updating HEAD in the submodule without touching its work tree doesn't seem like a good idea.  I think it will cause a lot more confusion when running `git status` which will show unexpected modified content for the submodule.
>
> Yes, we agree here.
>
>> Since I did not expect rebase to perform a submodule update, I was not surprised to see unstaged submodule changes when rebasing, but I did expect rebase to commit anything I had added to the index.
>
> Right.
>
> I'll have to add some tests for that case, but I doubt I'll manage that
> today. Until I can provide a complete patch, this diff should fix your
> problem (no, I did not test if that change is enough to fix the problem,
> but at least it does not break the test suite ;-):
>
> ---------------8<--------------
> diff --git a/git-rebase--interactive.sh b/git-rebase--interactive.sh
> index 5812222..4546749 100644
> --- a/git-rebase--interactive.sh
> +++ b/git-rebase--interactive.sh
> @@ -672,7 +672,7 @@ rearrange_squash () {
>   case "$action" in
>   continue)
>          # do we have anything to commit?
> -       if git diff-index --cached --quiet --ignore-submodules HEAD --
> +       if git diff-index --cached --quiet HEAD --
>          then
>                  : Nothing to commit -- skip this
>          else

This patch does indeed fix the problem I was seeing.


Thanks,

-- 
John