Possible bug: Git submodules can get into broken state

Possible bug: Git submodules can get into broken state

[Chris Wilson]

Hi all,

If your git repo's .gitmodules contains a URL that you don't have access 
to (for example you download someone else's code and it references a 
submodule using their writable git@github.com URL) then:

* git submodule init will add them to .git/config, with the wrong URLs.

* git submodule update will fail to check out the repos, leaving an empty 
directory for the first one, and nothing for the others.

This state is broken (wrong URLs in .git/config), and AFAIK there's 
nothing you can do to check out these submodules without either:

(a) manually hacking them out of .git/config, or

(b) doing "git submodule rm" and then "git checkout .gitmodules" to undo 
the damage to that file.

The procedure I tried, which I expected to work, was:

* git submodule sync (doesn't sync them, because the directories don't 
exist or don't contain a valid git repo?)

* git submodule init (ignores them, because they're already in 
.git/config?)

* git submodule update (still fails because the URL in .git/config is 
wrong).

The new deinit command may help, but for the wrong reasons. I don't want 
to have to deinit my modules every time in the fabric deployment script, 
just so that if they get into this state, they will get unbroken 
automatically.

It seems wrong to me that neither "git submodule init" nor "git submodule 
sync" will modify the URL in .git/config, if the submodule is not already 
checked out. I think I'd expect "git submodule init" to be idempotent, so 
it would update the URLs in .git/config if they already exist, just like 
it adds the URLs if they don't.

Any advice? Is this a real bug?

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: Possible bug: Git submodules can get into broken state

[Junio C Hamano]

Chris Wilson <chris+gitscm@aptivate.org> writes:

> If your git repo's .gitmodules contains a URL that you don't have
> access to (for example you download someone else's code and it
> references a submodule using their writable git@github.com URL) then:
>
> * git submodule init will add them to .git/config, with the wrong URLs.
>
> * git submodule update will fail to check out the repos, leaving an
> empty directory for the first one, and nothing for the others.
>
> This state is broken (wrong URLs in .git/config), and AFAIK there's
> nothing you can do to check out these submodules without either:
>
> (a) manually hacking them out of .git/config, or

I do not think updating the config is "hacking", but is a perfectly
normal thing to do for a submodule user who wants to use a custom
URL different from what is recorded in .gitmodules (even when the
URL in .gitmodules is _working_, you may have a closer mirror you
would prefer to use, for example).  It is how the configuration is
designed to be used, if I am not mistaken.

So I do not see any breakage here.

Re: Possible bug: Git submodules can get into broken state

[Jens Lehmann]

Am 04.04.2013 19:10, schrieb Chris Wilson:
> If your git repo's .gitmodules contains a URL that you don't have access to (for example you download someone else's code and it references a submodule using their writable git@github.com URL) then:
> 
> * git submodule init will add them to .git/config, with the wrong URLs.
> 
> * git submodule update will fail to check out the repos, leaving an empty directory for the first one, and nothing for the others.
> 
> This state is broken (wrong URLs in .git/config), and AFAIK there's nothing you can do to check out these submodules without either:
> 
> (a) manually hacking them out of .git/config, or

... or:

(c) Enter the correct URL in .git/config.

> (b) doing "git submodule rm" and then "git checkout .gitmodules" to undo the damage to that file.

Hmm ... that leaves your superproject dirty, right?

(d) Update the .gitmodules file to use the correct URL (you
    want to do a commit fixing that anyway, no? ;-) and do a
    "git submodule sync", which will copy the corrected URL
    into .git/config.


> The procedure I tried, which I expected to work, was:
> 
> * git submodule sync (doesn't sync them, because the directories don't exist or don't contain a valid git repo?)

No, because .gitmodules still contained the broken URL which a
sync then copies into .git/config again.

> * git submodule init (ignores them, because they're already in .git/config?)

Correct.

> * git submodule update (still fails because the URL in .git/config is wrong).

Sure.

> The new deinit command may help, but for the wrong reasons. I don't want to have to deinit my modules every time in the fabric deployment script, just so that if they get into this state, they will get unbroken automatically.

I doubt deinit will help here (except after running that you'll
be able to use "git submodule update" to populate the remaining
submodules) unless you fix the broken URL in .git/config or
.gitmodules.

> It seems wrong to me that neither "git submodule init" nor "git submodule sync" will modify the URL in .git/config, if the submodule is not already checked out. I think I'd expect "git submodule init" to be idempotent, so it would update the URLs in .git/config if they already exist, just like it adds the URLs if they don't.
> 
> Any advice? Is this a real bug?

Hmm, at first glance this looks like a pilot error. Maybe we could
update the documentation to help other users falling into that
trap or extend some commands to be a bit more helpful in such a
case, but it looks like the behavior you observed is documented
(while fixing the problem you observed isn't explicitly). Also an
option for "git submodule update" to continue even if it fails to
populate some submodules might help here.

Re: Possible bug: Git submodules can get into broken state

[Chris Wilson]

Hi Junio,

On Thu, 4 Apr 2013, Junio C Hamano wrote:

>> This state is broken (wrong URLs in .git/config), and AFAIK there's
>> nothing you can do to check out these submodules without either:
>>
>> (a) manually hacking them out of .git/config, or
>
> I do not think updating the config is "hacking", but is a perfectly
> normal thing to do for a submodule user who wants to use a custom
> URL different from what is recorded in .gitmodules (even when the
> URL in .gitmodules is _working_, you may have a closer mirror you
> would prefer to use, for example).  It is how the configuration is
> designed to be used, if I am not mistaken.

It may be possible, but there's no easy command to do it, especially in 
automated (fabric) deployment scripts. I do not want to write an awk/sed 
script to remove all the submodules from .git/config so that I can 
successfully run git submodule init again. What other choices do I have?

> So I do not see any breakage here.

I do, it seems bizarre that git submodule init can create this situation 
but cannot rectify it.

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: Possible bug: Git submodules can get into broken state

[Junio C Hamano]

Chris Wilson <chris@aptivate.org> writes:

>  I do not want to write an
> awk/sed script to remove all the submodules from .git/config ...

Don't do it then ;-)

I think "git config" was added exactly because people wanted to
customize their configuration from their scripts.

> I do, it seems bizarre that git submodule init can create this
> situation but cannot rectify it.

I do not see there is anything to rectify in your situation.

Whoever wrote .gitmodules may have supplied a URL you cannot access,
and the configuration mechanism is the way to use custom URL in
place of it.

Re: Possible bug: Git submodules can get into broken state

[Chris Wilson]

On Thu, 4 Apr 2013, Junio C Hamano wrote:

>> I do not want to write an awk/sed script to remove all the submodules 
>> from .git/config ...
>
> Don't do it then ;-)
>
> I think "git config" was added exactly because people wanted to
> customize their configuration from their scripts.

OK, I didn't know about git config. I will investigate that. But if the 
only automated way to fix the configuration in .git/config is by removing 
all submodules and running submodule init again, I still think that's a 
usability issue at least. If you won't accept it as a bug, would you 
consider it a feature request?

>> I do, it seems bizarre that git submodule init can create this 
>> situation but cannot rectify it.
>
> I do not see there is anything to rectify in your situation.
>
> Whoever wrote .gitmodules may have supplied a URL you cannot access,

They originally supplied a wrong URL (one that I can't access).

They fixed it by checking in a rectified .gitmodules file.

In the mean time, I ran git submodule init, and now I've ended up in a 
situation where git submodule update doesn't work, and there's no 
submodule command to fix it, so I have to remove the broken submodules 
from .git/config.

> and the configuration mechanism is the way to use custom URL in
> place of it.

I don't want to use a custom URL, I want to use the URL which is now in 
.gitmodules.

Could submodule init at least change the URLs of submodules which are not 
checked out? (e.g. because they couldn't be)?

Cheers, Chris.
-- 
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

Re: Possible bug: Git submodules can get into broken state

[Junio C Hamano]

Chris Wilson <chris+gitscm@aptivate.org> writes:

> They fixed it by checking in a rectified .gitmodules file.
>
> In the mean time, I ran git submodule init, and now I've ended up in a
> situation where git submodule update doesn't work, and there's no
> submodule command to fix it, so I have to remove the broken submodules
> from .git/config.
>
>> and the configuration mechanism is the way to use custom URL in
>> place of it.
>
> I don't want to use a custom URL, I want to use the URL which is now
> in .gitmodules.

Then don't call it a "custom URL" ;-).

Isn't your problem that the old, broken one is now in your config,
and you want to update that with a corrected one?  How you learned
which one is correct does not really matter---you may learned it out
of band by looking at upstream's gitweb, or by running "git fetch"
in the superproject again and looking at the updated .gitmodules
file.  The configuration mechanism has a way to update that entry.

> Could submodule init at least change the URLs of submodules which are
> not checked out? (e.g. because they couldn't be)?

Perhaps "submodule deinit" might be what you are looking for, but I
dunno.