[RFC] URL rewrite in .gitmodules

[RFC] URL rewrite in .gitmodules

[Lars Schneider]

Hi,

I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the "outside of company" repos. I know I can rewrite Git URLs with the git config "url.<base>.insteadOf" option. However, git configs are client specific. I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the
  very same hashes as defined by the "upstream" Open Source repos.

Is there yet another way in Git to change URLs of Submodules in the way I want it?

If not, what do you think about a patch that adds a "url" section similar to the one in git config to a .gitmodules file?

Example:
----------
[submodule "git"]
	path = git
        url=git://github.com/larsxschneider/git.git

[url "mycompany.com"]
        insteadOf = outside.com
----------

Thanks,
Lars

Re: [RFC] URL rewrite in .gitmodules

[Stefan Beller]

On Mon, Oct 19, 2015 at 12:28 PM, Lars Schneider
<larsxschneider@gmail.com> wrote:
> Hi,
>
> I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the "outside of company" repos. I know I can rewrite Git URLs with the git config "url.<base>.insteadOf" option. However, git configs are client specific.

I feel like this is working as intended. You only want to improve your
one client (say the buildbot) to not goto the open source site, while
the developer may do want to fetch from external sources ("Hey shiny
new code!";)


> I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the very same hashes as defined by the "upstream" Open Source repos.

You could carry a patch on top of the tip of the first submodule
re-pointing the nested submodule. This requires good workflows
available to deal with submodules though. (Fetch and merge or rebase,
git submodule update should be able to do that?)

>
> Is there yet another way in Git to change URLs of Submodules in the way I want it?
>
> If not, what do you think about a patch that adds a "url" section similar to the one in git config to a .gitmodules file?
>

So we have different kinds of git configs. within one repository, in
the home director (global to the one machine),
maybe you would want to have one "global" config on a network share,
such that every box in your company
reads that "company-wide" global config and acts upon that?

> Example:
> ----------
> [submodule "git"]
>         path = git
>         url=git://github.com/larsxschneider/git.git
>
> [url "mycompany.com"]
>         insteadOf = outside.com

Wouldn't that be better put into say a global git config instead of
repeating it for every submodule?

In case of the nested submodule you would need to carry the last lines
as an extra patch anyway
if this was done in the .gitmodules files? Or do you expect this to be
applied recursively (i.e. nested
submodules all the way down also substitute outside.com)


> ----------
>

Am I missing your point?

> Thanks,
> Lars--
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] URL rewrite in .gitmodules

[Lars Schneider]

On 20 Oct 2015, at 00:07, Stefan Beller <sbeller@google.com> wrote:

> On Mon, Oct 19, 2015 at 12:28 PM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> Hi,
>> 
>> I have a closed source Git repo which references an Open Source Git repo as Submodule. The Open Source Git repo references yet another Open Source repo as submodule. In order to avoid failing builds due to external services I mirrored the Open Source repos in my company network. That works great with the first level of Submodules. Unfortunately it does not work with the second level because the first level still references the "outside of company" repos. I know I can rewrite Git URLs with the git config "url.<base>.insteadOf" option. However, git configs are client specific.
> 
> I feel like this is working as intended. You only want to improve your
> one client (say the buildbot) to not goto the open source site, while
> the developer may do want to fetch from external sources ("Hey shiny
> new code!";)
Well, that's a good argument. However, our developers have usually no write access to these repos. If they want to push a commit then they need to fork the open source repo and change the submodule URL in the parent repo. I fear that this kind of process might overwhelm them and/or troubles them (changing Git submodules URLs has a few pitfalls). As a result they might be less inclined to make a contribution or - even worse - they copy the code in the parent repo, don't use Submodules and make no contribution at all. 


> 
>> I would prefer a solution that works without setup on any client. I also know that I could update the .gitmodules file in the Open Source repo on the first level. I also would prefer not to do this as I want to use the very same hashes as defined by the "upstream" Open Source repos.
> 
> You could carry a patch on top of the tip of the first submodule
> re-pointing the nested submodule. This requires good workflows
> available to deal with submodules though. (Fetch and merge or rebase,
> git submodule update should be able to do that?)
True. However, we have many Git beginners and I fear that this workflow would overwhelm them.


>> 
>> Is there yet another way in Git to change URLs of Submodules in the way I want it?
>> 
>> If not, what do you think about a patch that adds a "url" section similar to the one in git config to a .gitmodules file?
>> 
> 
> So we have different kinds of git configs. within one repository, in
> the home director (global to the one machine),
> maybe you would want to have one "global" config on a network share,
> such that every box in your company
> reads that "company-wide" global config and acts upon that?
That could actually work. The only downside I see is that the devs need to intentionally update their "company" git config. We have +4000 engineers and therefore I want to establish processes that are as easy and fault-tolerant as possible.  

> 
>> Example:
>> ----------
>> [submodule "git"]
>>        path = git
>>        url=git://github.com/larsxschneider/git.git
>> 
>> [url "mycompany.com"]
>>        insteadOf = outside.com
> 
> Wouldn't that be better put into say a global git config instead of
> repeating it for every submodule?
See answer above. The git config setup could be an obstacle.

> 
> In case of the nested submodule you would need to carry the last lines
> as an extra patch anyway
> if this was done in the .gitmodules files? Or do you expect this to be
> applied recursively (i.e. nested
> submodules all the way down also substitute outside.com)
Yes, my intention was to apply these recursively.


> Am I missing your point?
I don't think so :-)

Thanks,
Lars

Re: [RFC] URL rewrite in .gitmodules

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> If not, what do you think about a patch that adds a "url" section
> similar to the one in git config to a .gitmodules file?
>
> Example:
> ----------
> [submodule "git"]
> 	path = git
>         url=git://github.com/larsxschneider/git.git
>
> [url "mycompany.com"]
>         insteadOf = outside.com
> ----------

It is unclear to me if you are adding the last two (or three,
counting the blank before) lines to your company's private fork of
the opensource project, but if that is the case, then that would
defeat your earlier desire:

> ... I also would prefer not to do this as I want to use the
> very same hashes as defined by the "upstream" ...

wouldn't it?

I do not think this topic is specific to use of submodules.  If you
want to encourage your engineers to fetch from nearby mirrors you
maintain, you would want a forest of url.mine.insteadof=theirs for
the external repositories that matter to you specified by
everybody's $HOME/.gitconfig, and one way to do so would be to have
them use the configuration inclusion.  An item in your engineer
orientation material could tell them to add

	[include]
		path = /usr/local/etc/git/mycompany.urlrewrite

when they set up their "[user] name/email" in there.

And you can update /usr/local/etc/git/mycompany.urlrewrite as
needed.

Re: [RFC] URL rewrite in .gitmodules

[Lars Schneider]

On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:

> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>> If not, what do you think about a patch that adds a "url" section
>> similar to the one in git config to a .gitmodules file?
>> 
>> Example:
>> ----------
>> [submodule "git"]
>> 	path = git
>>        url=git://github.com/larsxschneider/git.git
>> 
>> [url "mycompany.com"]
>>        insteadOf = outside.com
>> ----------
> 
> It is unclear to me if you are adding the last two (or three,
> counting the blank before) lines to your company's private fork of
> the opensource project, but if that is the case, then that would
> defeat your earlier desire:
> 
>> ... I also would prefer not to do this as I want to use the
>> very same hashes as defined by the "upstream" ...
> 
> wouldn't it?
The last three lines are added to my companies closed source Git repo. In this example the company repo references git://github.com/larsxschneider/git.git as submodule. This submodule in turn references another submodule with a URL "outside.com". This is the URL I want to rewrite. Do you think this could be useful to others as well?


> I do not think this topic is specific to use of submodules.  If you
> want to encourage your engineers to fetch from nearby mirrors you
> maintain, you would want a forest of url.mine.insteadof=theirs for
> the external repositories that matter to you specified by
> everybody's $HOME/.gitconfig, and one way to do so would be to have
> them use the configuration inclusion.  An item in your engineer
> orientation material could tell them to add
> 
> 	[include]
> 		path = /usr/local/etc/git/mycompany.urlrewrite
> 
> when they set up their "[user] name/email" in there.
> 
> And you can update /usr/local/etc/git/mycompany.urlrewrite as
> needed.
Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.

Thanks,
Lars

Re: [RFC] URL rewrite in .gitmodules

[Stefan Beller]

On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:
>
>> Lars Schneider <larsxschneider@gmail.com> writes:
>>
>>> If not, what do you think about a patch that adds a "url" section
>>> similar to the one in git config to a .gitmodules file?
>>>
>>> Example:
>>> ----------
>>> [submodule "git"]
>>>      path = git
>>>        url=git://github.com/larsxschneider/git.git
>>>
>>> [url "mycompany.com"]
>>>        insteadOf = outside.com
>>> ----------
>>
>> It is unclear to me if you are adding the last two (or three,
>> counting the blank before) lines to your company's private fork of
>> the opensource project, but if that is the case, then that would
>> defeat your earlier desire:
>>
>>> ... I also would prefer not to do this as I want to use the
>>> very same hashes as defined by the "upstream" ...
>>
>> wouldn't it?
> The last three lines are added to my companies closed source Git repo. In this example the company repo references git://github.com/larsxschneider/git.git as submodule. This submodule in turn references another submodule with a URL "outside.com". This is the URL I want to rewrite. Do you think this could be useful to others as well?
>
>
>> I do not think this topic is specific to use of submodules.  If you
>> want to encourage your engineers to fetch from nearby mirrors you
>> maintain, you would want a forest of url.mine.insteadof=theirs for
>> the external repositories that matter to you specified by
>> everybody's $HOME/.gitconfig, and one way to do so would be to have
>> them use the configuration inclusion.  An item in your engineer
>> orientation material could tell them to add
>>
>>       [include]
>>               path = /usr/local/etc/git/mycompany.urlrewrite
>>
>> when they set up their "[user] name/email" in there.
>>
>> And you can update /usr/local/etc/git/mycompany.urlrewrite as
>> needed.
> Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.

How do you distribute new copies of Git to your engineers?
Maybe you could ship them a version which has the "include" line
already builtin as default? So your distributed copy of Git
would not just check the default places for configs, but also
some complied in /net/share/mycompany.gitconfig

>
> Thanks,
> Lars
>

Re: [RFC] URL rewrite in .gitmodules

[Jens Lehmann]

Am 26.10.2015 um 17:34 schrieb Stefan Beller:
> On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider <larsxschneider@gmail.com> wrote:
>> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:
>>> I do not think this topic is specific to use of submodules.  If you
>>> want to encourage your engineers to fetch from nearby mirrors you
>>> maintain, you would want a forest of url.mine.insteadof=theirs for
>>> the external repositories that matter to you specified by
>>> everybody's $HOME/.gitconfig, and one way to do so would be to have
>>> them use the configuration inclusion.  An item in your engineer
>>> orientation material could tell them to add
>>>
>>>        [include]
>>>                path = /usr/local/etc/git/mycompany.urlrewrite
>>>
>>> when they set up their "[user] name/email" in there.
>>>
>>> And you can update /usr/local/etc/git/mycompany.urlrewrite as
>>> needed.
>> Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.
>
> How do you distribute new copies of Git to your engineers?
> Maybe you could ship them a version which has the "include" line
> already builtin as default? So your distributed copy of Git
> would not just check the default places for configs, but also
> some complied in /net/share/mycompany.gitconfig

Which is just what we do at $DAYJOB, that way you can easily
distribute all kinds of settings, customizations and hooks
company-wide.

Re: [RFC] URL rewrite in .gitmodules

[Lars Schneider]

On 26 Oct 2015, at 17:52, Jens Lehmann <Jens.Lehmann@web.de> wrote:

> Am 26.10.2015 um 17:34 schrieb Stefan Beller:
>> On Sun, Oct 25, 2015 at 8:12 AM, Lars Schneider <larsxschneider@gmail.com> wrote:
>>> On 20 Oct 2015, at 19:33, Junio C Hamano <gitster@pobox.com> wrote:
>>>> I do not think this topic is specific to use of submodules.  If you
>>>> want to encourage your engineers to fetch from nearby mirrors you
>>>> maintain, you would want a forest of url.mine.insteadof=theirs for
>>>> the external repositories that matter to you specified by
>>>> everybody's $HOME/.gitconfig, and one way to do so would be to have
>>>> them use the configuration inclusion.  An item in your engineer
>>>> orientation material could tell them to add
>>>> 
>>>>       [include]
>>>>               path = /usr/local/etc/git/mycompany.urlrewrite
>>>> 
>>>> when they set up their "[user] name/email" in there.
>>>> 
>>>> And you can update /usr/local/etc/git/mycompany.urlrewrite as
>>>> needed.
>>> Oh nice, I didn't know about "include". However, as mentioned to Stefan in this thread, I fear that our engineers will miss that. I would prefer a solution that does not need any additional setup. Therefore the suggestion to add rewrites in the .gitmodules file.
>> 
>> How do you distribute new copies of Git to your engineers?
>> Maybe you could ship them a version which has the "include" line
>> already builtin as default? So your distributed copy of Git
>> would not just check the default places for configs, but also
>> some complied in /net/share/mycompany.gitconfig
> 
> Which is just what we do at $DAYJOB, that way you can easily
> distribute all kinds of settings, customizations and hooks
> company-wide.

That's a very good idea. I will try to establish this practice, too.

Thanks,
Lars