Unexpected behaviour after removing submodule

Unexpected behaviour after removing submodule

[Nicolas Morey-Chaisemartin]

Hi,

At work, we use a lot of submodules (several levels of submodules actually).
As we also work with development branches, we use scripts to resync the whole checked-out tree (mainly in automated integration)

We recently run across an issue where a branch (dev) contained a submodule while it had been removed in master and the files were imported in a subdirectory with the same name (probably using git-subtree).

Basically:

On dev:
* top/refs(submodule)/file1
On master:
* top/refs(dir)/file1

Outside the fact that it is quite hard to move from one branch to the other while having a perfectly clean tree checked out underneath, we manage to end up into a weird (invalid) state
that was neither clearly described nor "easy" to cleanup (using standard git clean/checkout/reset commands).

While I cannot explain how we got in this state, here is a small test-case that produce the same results:
------
mkdir folder1
cd folder1
git init
echo "Ooops" > file
git add file
git commit -m "Add file"
cd ..
mkdir folder2
cd folder2
git init
mkdir folder1
echo "Ooops" > folder1/file
git add folder1/file
git commit -m "Add file again"
git checkout -b branch
cp -R ../folder1/.git ./folder1
------
The 'cp' just seems pointless but with the submodule described as above we manage to end up in a similar state.
In this state, when being in folder2, git status reports nothing. Dev branch is checked out and everything looks great.

However if you change dir to folder2/folder1, while still being inside folder2, git thinks (because of the .git dir) that you are actually on master branch of folder1 repository.
Which mean that if you happen to commit from a subdirectory, you may easily end-up committing in another repository than the one expected.

The issue is, there is no way from folder2 to see that something "wrong" is going on inside your tree!
As we manage to reach this state using only standard git commands (I'll try to reproduce it) with submodules, and this being part of an automated flow, it is quite worrying.
We may actually be committing in the wrong repo and pushing the wrong things around.

Is there or should there be a way to look for such issues? And is this an expected behaviour?

Thanks in advance

Nicolas

Re: Unexpected behaviour after removing submodule

[Jens Lehmann]

Am 25.10.2012 17:06, schrieb Nicolas Morey-Chaisemartin:
> At work, we use a lot of submodules (several levels of submodules actually).
> As we also work with development branches, we use scripts to resync the whole checked-out tree (mainly in automated integration)
> 
> We recently run across an issue where a branch (dev) contained a submodule while it had been removed in master and the files were imported in a subdirectory with the same name (probably using git-subtree).
> 
> Basically:
> 
> On dev:
> * top/refs(submodule)/file1
> On master:
> * top/refs(dir)/file1
> 
> Outside the fact that it is quite hard to move from one branch to the other while having a perfectly clean tree checked out underneath, we manage to end up into a weird (invalid) state
> that was neither clearly described nor "easy" to cleanup (using standard git clean/checkout/reset commands).

<snipped example without submodules>

> The issue is, there is no way from folder2 to see that something "wrong" is going on inside your tree!
> As we manage to reach this state using only standard git commands (I'll try to reproduce it) with submodules, and this being part of an automated flow, it is quite worrying.
> We may actually be committing in the wrong repo and pushing the wrong things around.
> 
> Is there or should there be a way to look for such issues? And is this an expected behaviour?

Unfortunately this is how things work at the moment. If you remove a
submodule its work tree will currently stay around nonetheless. And
when you replace it with a directory containing files tracked by git,
things start to get really weird when you do checkouts crossing that
conversion.

But the solution to that problem is coming closer. The first step was
to move the .git directory out of the submodule's work tree, so that
we are able to remove it without loosing any history. The next step
will be to enable git to remove and re-add submodules on checkout. The
"remove a submodule" part already works in my - still experimental -
recursive_submodule_checkout branch at my github repo. The thing that
is still missing - apart from tests ;) - is that checkout has to learn
to look into the to-be-checked-out .gitmodules to be able to populate
a re-appearing submodule. I plan to add that for git fetch first (so
it can fetch submodules the user showed interest in but which aren't
currently checked out) and after that I'll reuse it for checkout.

But that's still some work to do and will take some time ...

Re: Unexpected behaviour after removing submodule

[Nicolas Morey-Chaisemartin]

On 26/10/12 00:47, Jens Lehmann wrote:
> Am 25.10.2012 17:06, schrieb Nicolas Morey-Chaisemartin:
>> At work, we use a lot of submodules (several levels of submodules actually).
>> As we also work with development branches, we use scripts to resync the whole checked-out tree (mainly in automated integration)
>>
>> We recently run across an issue where a branch (dev) contained a submodule while it had been removed in master and the files were imported in a subdirectory with the same name (probably using git-subtree).
>>
>> Basically:
>>
>> On dev:
>> * top/refs(submodule)/file1
>> On master:
>> * top/refs(dir)/file1
>>
>> Outside the fact that it is quite hard to move from one branch to the other while having a perfectly clean tree checked out underneath, we manage to end up into a weird (invalid) state
>> that was neither clearly described nor "easy" to cleanup (using standard git clean/checkout/reset commands).
> <snipped example without submodules>
>
>> The issue is, there is no way from folder2 to see that something "wrong" is going on inside your tree!
>> As we manage to reach this state using only standard git commands (I'll try to reproduce it) with submodules, and this being part of an automated flow, it is quite worrying.
>> We may actually be committing in the wrong repo and pushing the wrong things around.
>>
>> Is there or should there be a way to look for such issues? And is this an expected behaviour?
> Unfortunately this is how things work at the moment. If you remove a
> submodule its work tree will currently stay around nonetheless. And
> when you replace it with a directory containing files tracked by git,
> things start to get really weird when you do checkouts crossing that
> conversion.
>
> But the solution to that problem is coming closer. The first step was
> to move the .git directory out of the submodule's work tree, so that
> we are able to remove it without loosing any history. The next step
> will be to enable git to remove and re-add submodules on checkout. The
> "remove a submodule" part already works in my - still experimental -
> recursive_submodule_checkout branch at my github repo. The thing that
> is still missing - apart from tests ;) - is that checkout has to learn
> to look into the to-be-checked-out .gitmodules to be able to populate
> a re-appearing submodule. I plan to add that for git fetch first (so
> it can fetch submodules the user showed interest in but which aren't
> currently checked out) and after that I'll reuse it for checkout.
>
> But that's still some work to do and will take some time ...
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

That's what I figured. I just wished there was at least a way to see that there's a .git directory that does not really belong.
To check that now we'd have to do a git-ls tree and look for .git that shouldn't be here but with the amount of file we have, a shell script doing that would take ages !

We'll wait for a more submodule aware version of checkout then !

Thanks

Nicolas