[Question] Switching the URI from SSH to HTTPS for submodules

[Question] Switching the URI from SSH to HTTPS for submodules

[rsbecker]

This is a follow-up to the conversation at #git-devel Standup today.

We encountered the following situation. Our normal clone practice is to use
SSH URIs. So, the clone is straightforward with the main (8 year old) repo
of the style:

git@bitbucket.org:project/repo.git

with submodules referenced as:

git@bitbucket.org:project/module.git

When in SSH mode, clones are simple with --recurse-submodules doing what we
want. However, we had to clone on a system where SSH was locked down and we
could only use HTTPS. The form of the URIs changed rather radically:

https://user@bitbucket.org/project/repo.git

which caused the submodule clone not to work. Fortunately, the number of
submodules was low, so a manual edit of .gitmodules (with update-index
--assume-unchanged) and .git/modules and .git/config was not onerous.
However, we should be able to redirect the configuration without having to
do that. Our directory references are all relative to the main repo, so
those did not have to change. We still use SSH for most work with the repo
but needed to get source onto a test system for debugging. 

I would like to put together a best practice section in
Documentation/gitsubmodules.txt on the subject, but really don't think I did
this *temporary* change the most effective way. I'm looking for better
practices than I used - I'm sure there is at least one. There may be code
changes involved, particularly since .gitmodules is tracked and contains the
full URI (maybe it should not). One option we also explored was explicitly
specifying the URI for each submodule using set-url, which works but it
seems like there should be something in clone that redirects the part of the
URI relative to the main repo - although that would only work in the special
case where the submodules have the same host and project as the main repo.

-Randall

Re: [Question] Switching the URI from SSH to HTTPS for submodules

[Robert Coup]

Hi Randall,

On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:
>
> git@bitbucket.org:project/module.git
>
> When in SSH mode, clones are simple with --recurse-submodules doing what we
> want. However, we had to clone on a system where SSH was locked down and we
> could only use HTTPS. The form of the URIs changed rather radically:
>
> https://user@bitbucket.org/project/repo.git
>
> I'm looking for better practices than I used - I'm sure there is at least one.

AFAIK the existing `url.<base>.insteadOf` config option[1] deals with this...

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf

    $ git config --global url."https://user@bitbucket.org/".insteadOf
"git@bitbucket.org:"

If you don't want to set it globally (not a throwaway CI environment?)
then you can do it as a one-off:

    $ git -c url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
clone --recurse-submodules git@bitbucket.org:repo/project.git

But it isn't persisted into your repo config then, so subsequent
fetches won't work. You'd need to persist it using something like:

    $ git config url."https://user@bitbucket.org/".insteadOf
"git@bitbucket.org:"
    $ git submodule foreach --recursive 'git config
url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '

Maybe there's an opportunity to make that part easier?

Rob :)

RE: [Question] Switching the URI from SSH to HTTPS for submodules

[rsbecker]

On December 6, 2021 6:54 PM, Robert Coup wrote:
> On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:
> >
> > git@bitbucket.org:project/module.git
> >
> > When in SSH mode, clones are simple with --recurse-submodules doing
> > what we want. However, we had to clone on a system where SSH was
> > locked down and we could only use HTTPS. The form of the URIs changed
> rather radically:
> >
> > https://user@bitbucket.org/project/repo.git
> >
> > I'm looking for better practices than I used - I'm sure there is at least one.
> 
> AFAIK the existing `url.<base>.insteadOf` config option[1] deals with this...
> 
> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-
> urlltbasegtinsteadOf
> 
>     $ git config --global url."https://user@bitbucket.org/".insteadOf
> "git@bitbucket.org:"
> 
> If you don't want to set it globally (not a throwaway CI environment?) then
> you can do it as a one-off:
> 
>     $ git -c url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
> clone --recurse-submodules git@bitbucket.org:repo/project.git
> 
> But it isn't persisted into your repo config then, so subsequent fetches won't
> work. You'd need to persist it using something like:
> 
>     $ git config url."https://user@bitbucket.org/".insteadOf
> "git@bitbucket.org:"
>     $ git submodule foreach --recursive 'git config
> url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '
> 
> Maybe there's an opportunity to make that part easier?

Well, I gave this a shot. The mapping did not appear to work - I tried a few combinations. I was left with the original URIs. Note that I also had to do 

$ git submodule init

which reported the wrong URIs

$ git submodule update

which obviously failed since they were still using SSH.

I think there may be something not working correctly with the insteadOf operator. This is easily reproduceable. 

-Randall

Re: [Question] Switching the URI from SSH to HTTPS for submodules

[Fabian Stelzer]

On 07.12.2021 09:50, rsbecker@nexbridge.com wrote:
>On December 6, 2021 6:54 PM, Robert Coup wrote:
>> On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:
>> >
>> > git@bitbucket.org:project/module.git
>> >
>> > When in SSH mode, clones are simple with --recurse-submodules doing
>> > what we want. However, we had to clone on a system where SSH was
>> > locked down and we could only use HTTPS. The form of the URIs changed
>> rather radically:
>> >
>> > https://user@bitbucket.org/project/repo.git
>> >
>> > I'm looking for better practices than I used - I'm sure there is at least one.
>>
>> AFAIK the existing `url.<base>.insteadOf` config option[1] deals with this...
>>
>> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-
>> urlltbasegtinsteadOf
>>
>>     $ git config --global url."https://user@bitbucket.org/".insteadOf
>> "git@bitbucket.org:"
>>
>> If you don't want to set it globally (not a throwaway CI environment?) then
>> you can do it as a one-off:
>>
>>     $ git -c url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
>> clone --recurse-submodules git@bitbucket.org:repo/project.git
>>
>> But it isn't persisted into your repo config then, so subsequent fetches won't
>> work. You'd need to persist it using something like:
>>
>>     $ git config url."https://user@bitbucket.org/".insteadOf
>> "git@bitbucket.org:"
>>     $ git submodule foreach --recursive 'git config
>> url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '
>>
>> Maybe there's an opportunity to make that part easier?
>
>Well, I gave this a shot. The mapping did not appear to work - I tried a few combinations. I was left with the original URIs. Note that I also had to do
>
>$ git submodule init
>
>which reported the wrong URIs
>
>$ git submodule update
>
>which obviously failed since they were still using SSH.
>
>I think there may be something not working correctly with the insteadOf operator. This is easily reproduceable.
>

I ran into this as well some time ago. For submodules this only works if you 
put it into your global config since the parent projects config is not 
relevant to the submodule.

There's some discussion in the archive about this as well:
https://lore.kernel.org/git/404d109f-e5a7-85a3-e64c-ab1b21c3045d@durchholz.org/

RE: [Question] Switching the URI from SSH to HTTPS for submodules

[rsbecker]

On December 8, 2021 7:52 AM, Fabian Stelzer wrote:
> On 07.12.2021 09:50, rsbecker@nexbridge.com wrote:
> >On December 6, 2021 6:54 PM, Robert Coup wrote:
> >> On Mon, 6 Dec 2021 at 23:10, <rsbecker@nexbridge.com> wrote:
> >> >
> >> > git@bitbucket.org:project/module.git
> >> >
> >> > When in SSH mode, clones are simple with --recurse-submodules doing
> >> > what we want. However, we had to clone on a system where SSH was
> >> > locked down and we could only use HTTPS. The form of the URIs
> >> > changed
> >> rather radically:
> >> >
> >> > https://user@bitbucket.org/project/repo.git
> >> >
> >> > I'm looking for better practices than I used - I'm sure there is at least
> one.
> >>
> >> AFAIK the existing `url.<base>.insteadOf` config option[1] deals with
> this...
> >>
> >> [1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-
> >> urlltbasegtinsteadOf
> >>
> >>     $ git config --global url."https://user@bitbucket.org/".insteadOf
> >> "git@bitbucket.org:"
> >>
> >> If you don't want to set it globally (not a throwaway CI
> >> environment?) then you can do it as a one-off:
> >>
> >>     $ git -c
> url."https://user@bitbucket.org/".insteadOf="git@bitbucket.org:"
> >> clone --recurse-submodules git@bitbucket.org:repo/project.git
> >>
> >> But it isn't persisted into your repo config then, so subsequent
> >> fetches won't work. You'd need to persist it using something like:
> >>
> >>     $ git config url."https://user@bitbucket.org/".insteadOf
> >> "git@bitbucket.org:"
> >>     $ git submodule foreach --recursive 'git config
> >> url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:" '
> >>
> >> Maybe there's an opportunity to make that part easier?
> >
> >Well, I gave this a shot. The mapping did not appear to work - I tried
> >a few combinations. I was left with the original URIs. Note that I also
> >had to do
> >
> >$ git submodule init
> >
> >which reported the wrong URIs
> >
> >$ git submodule update
> >
> >which obviously failed since they were still using SSH.
> >
> >I think there may be something not working correctly with the insteadOf
> operator. This is easily reproduceable.
> >
> 
> I ran into this as well some time ago. For submodules this only works if you
> put it into your global config since the parent projects config is not relevant to
> the submodule.
> 
> There's some discussion in the archive about this as well:
> https://lore.kernel.org/git/404d109f-e5a7-85a3-e64c-
> ab1b21c3045d@durchholz.org/

The minimal procedure that I got to work is:

$ git config --global url."https://user@bitbucket.org/".insteadOf "git@bitbucket.org:"
$ git clone --recurse-submodules https://user@bitbucket.org/project/repo.git 

This leaves .gitmodules untouched and contains the original URIs, so the status is clean and there is no real risk of someone pushing an unwanted URI change back to the upstream repo. I don't see a major downside to leaving the redirect in global as this would be useful for subsequent clones. Any scripting that might depend on .gitmodules being correct would break - not in my case though - to that should probably be noted.

I think I have enough to document this consideration in Documentation/gitsubmodules.txt. Thanks everyone.
-Randall