url.<base>.insteadOf vs. submodules

url.<base>.insteadOf vs. submodules

[Toolforger]

Hi all,

I am trying to make url.<base>.insteadOf work on the URLs inside 
.gitmodules, but it won't work (applying it to the repo itself works 
fine, to the config setting seems to be fine).

I do not want to modify .gitmodules: It is maintained upstream.

I cannot simply reconfigure submodule.<module>.url: the Configure script 
(regularly called during each compile) does
   git submodule sync
   git submodule update --init
I could tell upstream to change these commands if I can make a good 
argument; for them, it is relevant that they can change the submodule 
URL inside .gitmodule and have it "just work" for everybody downstream.

My own use case is that I want to be able to work with various 
experimental local clones even if I do not have Internet access.
I'm all ears if there's a way to do this without using insteadOf.


Here are the relevant two lines from the output of "git config -l" 
(after "git submodule init"):

url./home/jo/Projekte/perl6/bare-repos.insteadof=https://github.com
submodule.3rdparty/dynasm.url=https://github.com/MoarVM/dynasm.git


Here is what "git submodule update" does:

Cloning into '3rdparty/dyncall'...
fatal: unable to access 'https://github.com/MoarVM/dyncall.git/': Could 
not resolve host: github.com
fatal: clone of 'https://github.com/MoarVM/dyncall.git' into submodule 
path '3rdparty/dyncall' failed


Any help appreciated!

Regards,
Jo

Re: url.<base>.insteadOf vs. submodules

[Jeff King]

On Sun, Feb 19, 2017 at 10:12:28PM +0100, Toolforger wrote:

> I am trying to make url.<base>.insteadOf work on the URLs inside
> .gitmodules, but it won't work (applying it to the repo itself works fine,
> to the config setting seems to be fine).

The submodule operations happen in their own processes, and do not look
at the config of the parent repo. Are you setting the config in
.git/config of the super-project?

I don't know if there plans to make that work, but one workaround is to
set the config in ~/.gitconfig.

-Peff

Re: url.<base>.insteadOf vs. submodules

[Toolforger]

On 20.02.2017 10:01, Jeff King wrote:
> On Sun, Feb 19, 2017 at 10:12:28PM +0100, Toolforger wrote:
>
>> I am trying to make url.<base>.insteadOf work on the URLs inside
>> .gitmodules, but it won't work (applying it to the repo itself works fine,
>> to the config setting seems to be fine).
>
> The submodule operations happen in their own processes, and do not look
> at the config of the parent repo.

Ah, then we have a docbug.
git help config has this to say:

url.<base>.insteadOf
     Any URL that starts with this value will be rewritten to start,
     instead, with <base>.

The "Any" here is wrong, it would be "any except submodule" (possibly 
other exceptions).

 > Are you setting the config in
> .git/config of the super-project?

Exactly.
My thinking was that since the submodule URLs are specified in the super 
project's .gitmodules, that setting should apply.

> I don't know if there plans to make that work,

It would certainly help me out, though I guess it's going to be too late 
for my current project :-)

 > but one workaround is to set the config in ~/.gitconfig.

No can do - that's under version control.
My personal setup does not belong there I think ;-)

I am currently trying to write a shell script that
- does git submodule init
- pulls submodule configuration out of git config -l
- configures each submodule with insteadOf
It fits with my workflow because setting up the repositories is going to 
be done via script anyway.
I'm neither a shell nor a git expert, so any advice still appreciated.

Regards,
Jo

Re: url.<base>.insteadOf vs. submodules

[Jeff King]

On Mon, Feb 20, 2017 at 09:31:40PM +0100, Toolforger wrote:

> > The submodule operations happen in their own processes, and do not look
> > at the config of the parent repo.
> 
> Ah, then we have a docbug.
> git help config has this to say:
> 
> url.<base>.insteadOf
>     Any URL that starts with this value will be rewritten to start,
>     instead, with <base>.
> 
> The "Any" here is wrong, it would be "any except submodule" (possibly other
> exceptions).

I'm not sure that "any" is wrong here. Repository-specific config does
not cross repository boundaries. That applies to this config value, and
to all the others, too (e.g., if you set "diff.renames" in the
super-project, it would not have an effect in the submodule).

I think if there is a doc bug, it is that the repo boundary between the
submodule and the super-project is not made more clear.

That said, I do think it would be a useful feature for the super-project
to rewrite URLs before handing them off to the submodule. But I do not
really work on submodules nor use them myself, so there may be
complications.

I suppose you could argue that failing to rewrite violates the "any" in
the quoted text. It doesn't say when the rewriting occurs, but it is
essentially "when the URL is accessed". So the super-project feeds the
raw URL to the submodule `git clone`, which then applies any URL
rewriting.

> > but one workaround is to set the config in ~/.gitconfig.
> 
> No can do - that's under version control.
> My personal setup does not belong there I think ;-)

I'm not sure I understand. You have a project policy to use certain
URLs. But you, the user, want to override that. Why isn't the
user-specific config file the right place to put that?

(I think there _is_ a mismatch, in that the change is specific not just
to your user, but to the repo. So you would not want to rewrite other
references to the same URL in other repos. But that does not seem to be
your objection).

-Peff

Re: url.<base>.insteadOf vs. submodules

[Toolforger]

On 20.02.2017 21:52, Jeff King wrote:
 > I think if there is a doc bug, it is that the repo boundary between the
 > submodule and the super-project is not made more clear.

It's not mentioned anywhere I'm aware of, particularly not on the 
insteadOf docs.

 > That said, I do think it would be a useful feature for the super-project
 > to rewrite URLs before handing them off to the submodule. But I do not
 > really work on submodules nor use them myself, so there may be
 > complications.

Agreed.

 > I suppose you could argue that failing to rewrite violates the "any" in
 > the quoted text. It doesn't say when the rewriting occurs, but it is
 > essentially "when the URL is accessed". So the super-project feeds the
 > raw URL to the submodule `git clone`, which then applies any URL
 > rewriting.



 >>> but one workaround is to set the config in ~/.gitconfig.
 >>
 >> No can do - that's under version control.
 >> My personal setup does not belong there I think ;-)
 >
 > I'm not sure I understand. You have a project policy to use certain
 > URLs. But you, the user, want to override that. Why isn't the
 > user-specific config file the right place to put that?

Ah right, I mistook ~/ for "project root" instead of "home dir".
Sorry for the confusion.

 > (I think there _is_ a mismatch, in that the change is specific not just
 > to your user, but to the repo. So you would not want to rewrite other
 > references to the same URL in other repos.

Indeed, and that's actually a problem.

The setup I'm aiming for is
   github -> local bare repo -> local clones with worktrees

If I place insteadOf rules in ~/.gitconfig, I will be unable to pull 
from github to my local bare repos.
Mmm... I could try to undo the insteadOf configuration from ~/.gitconfig 
in the local bare repos. Not sure whether I have to redirect from the 
github URL to itself.

Downside is that I'll have to remember to modify ~/.gitconfig whenever 
the upstream project changes its dependencies. Or whenever I want to 
reorganize my local project directory structure.
It's not totally out of the window, but right now it does not seem very 
attractive to me, and it's certainly not a good solution for everyone.

Regards,
Jo

Re: url.<base>.insteadOf vs. submodules

[Jeff King]

On Tue, Feb 21, 2017 at 06:11:51AM +0100, Toolforger wrote:

> > I'm not sure I understand. You have a project policy to use certain
> > URLs. But you, the user, want to override that. Why isn't the
> > user-specific config file the right place to put that?
> 
> Ah right, I mistook ~/ for "project root" instead of "home dir".
> Sorry for the confusion.

Ah, OK, that makes more sense.

> > (I think there _is_ a mismatch, in that the change is specific not just
> > to your user, but to the repo. So you would not want to rewrite other
> > references to the same URL in other repos.
> 
> Indeed, and that's actually a problem.
> 
> The setup I'm aiming for is
>   github -> local bare repo -> local clones with worktrees
> 
> If I place insteadOf rules in ~/.gitconfig, I will be unable to pull from
> github to my local bare repos.
> Mmm... I could try to undo the insteadOf configuration from ~/.gitconfig in
> the local bare repos. Not sure whether I have to redirect from the github
> URL to itself.

Yeah, I think you would probably have to do a redirect-to-self to
override the global one.

At one point we discussed having conditional-config that would kick in
based on path-matching. I think it would be another way to do what you
want, but there's nothing merged.

I think anything involving ~/.gitconfig is basically a hack, though.
What you really want is for submodules to better support your
URL-rewriting case, and that's not an unreasonable thing to want.

We'll see if the submodule folks have any ideas on how to implement
that.

-Peff

Re: url.<base>.insteadOf vs. submodules

[Stefan Beller]

On Mon, Feb 20, 2017 at 11:06 PM, Jeff King <peff@peff.net> wrote:
>
> We'll see if the submodule folks have any ideas on how to implement
> that.
>

So from reading your discussion, the user expectation is to have
`git submodule {init, update --init, sync}`
to pay attention to url.<base>.insteadOf when setting up the
submodule.<name>.URL, such that the modified URL is used for the
initial clone of the submodule (and hence any subsequent usage within
the submodule).

That sounds like a good idea to me.

Two caveates:

* After running `git submodule init`, you change url.<base>.insteadOf
  in the superproject. How do we need to word the documentation to
  have users expecting this change doesn't affect submodules?
  (See above Any vs. "Any except (initialized) submodules")

* So with the point above the insteadOf config only applies to the
  init/sync process, (i.e. once in time, ideally).
  Is that confusing or actually simplifying the submodule workflow?

Thanks,
Stefan

Re: url.<base>.insteadOf vs. submodules

[Jeff King]

On Tue, Feb 21, 2017 at 10:19:38AM -0800, Stefan Beller wrote:

> On Mon, Feb 20, 2017 at 11:06 PM, Jeff King <peff@peff.net> wrote:
> >
> > We'll see if the submodule folks have any ideas on how to implement
> > that.
> >
> 
> So from reading your discussion, the user expectation is to have
> `git submodule {init, update --init, sync}`
> to pay attention to url.<base>.insteadOf when setting up the
> submodule.<name>.URL, such that the modified URL is used for the
> initial clone of the submodule (and hence any subsequent usage within
> the submodule).

Yeah, that was what I was envisioning.

> Two caveates:
> 
> * After running `git submodule init`, you change url.<base>.insteadOf
>   in the superproject. How do we need to word the documentation to
>   have users expecting this change doesn't affect submodules?
>   (See above Any vs. "Any except (initialized) submodules")

Good question.

I guess one answer is that this is the wrong approach entirely, and the
right one is something like: submodules should understand that they are
part of a superproject, and respect some whitelisted set of config from
the superproject .git/config file.

The second half is pretty easy to do (use git_config_from_file on the
super-project's $GIT_DIR/config, and pass a callback which filters the
keys before passing them along to the real callback).

I'm not sure about the first half (submodules know about their
superproject), though.

> * So with the point above the insteadOf config only applies to the
>   init/sync process, (i.e. once in time, ideally).
>   Is that confusing or actually simplifying the submodule workflow?

Not sure. That's why I asked you. :)

One other caveat: I'm not sure if we do insteadOf recursively, but it
may be surprising to the child "git clone" that we've already applied
the insteadOf rewriting (especially if the rules are coming from
~/.gitconfig and may be applied twice).

-Peff

Re: url.<base>.insteadOf vs. submodules

[Stefan Beller]

On Tue, Feb 21, 2017 at 3:00 PM, Jeff King <peff@peff.net> wrote:
> On Tue, Feb 21, 2017 at 10:19:38AM -0800, Stefan Beller wrote:
>
>> On Mon, Feb 20, 2017 at 11:06 PM, Jeff King <peff@peff.net> wrote:
>> >
>> > We'll see if the submodule folks have any ideas on how to implement
>> > that.
>> >
>>
>> So from reading your discussion, the user expectation is to have
>> `git submodule {init, update --init, sync}`
>> to pay attention to url.<base>.insteadOf when setting up the
>> submodule.<name>.URL, such that the modified URL is used for the
>> initial clone of the submodule (and hence any subsequent usage within
>> the submodule).
>
> Yeah, that was what I was envisioning.
>
>> Two caveates:
>>
>> * After running `git submodule init`, you change url.<base>.insteadOf
>>   in the superproject. How do we need to word the documentation to
>>   have users expecting this change doesn't affect submodules?
>>   (See above Any vs. "Any except (initialized) submodules")
>
> Good question.
>
> I guess one answer is that this is the wrong approach entirely, and the
> right one is something like: submodules should understand that they are
> part of a superproject, and respect some whitelisted set of config from
> the superproject .git/config file.

This would break one of the core assumptions that submodules
are "independent" repos.

The way of action is a one way street:
* The superproject is aware of the submodule and when you invoke a
command on the superproject, you may mess around with the submodule,
e.g. update/remove it; absorb its git directory.
* The submodule is "just" a repository with weird .git link file and a
  respective core.worktree setup. Currently it doesn't know if it is
  guided by a superproject.


Though I do not know if this is actually a good assumption.
e.g. "[PATCH v2] git-prompt.sh: add submodule indicator"
https://public-inbox.org/git/1486075892-20676-2-git-send-email-email@benjaminfuchs.de/
really had trouble in the first version to nail down how to tell you are in
a submodule, but people want to know that.

>
> The second half is pretty easy to do (use git_config_from_file on the
> super-project's $GIT_DIR

There goes the "pretty easy"; currently there is no concept to find out
the existence of a super-project.

> /config, and pass a callback which filters the
> keys before passing them along to the real callback).
>
> I'm not sure about the first half (submodules know about their
> superproject), though.

Maybe we need to change that fundamental assumption.
So a more sophisticated way (thinking long term here) would be
to include the superprojects config file (with exceptions), and that
config file has more priority than e.g. the ~/.gitconfig file, but less
than the submodules own $GIT_DIR/config file.
Then a setting like the url rewriting would be "inherited" by the
submodule, with the option to overwrite the default as given by the
superproject.

>
>> * So with the point above the insteadOf config only applies to the
>>   init/sync process, (i.e. once in time, ideally).
>>   Is that confusing or actually simplifying the submodule workflow?
>
> Not sure. That's why I asked you. :)

I think that would be ok. With the idea of inheriting the superprojects
config, we allow for not storing the rewritten url, so the submodule
handling is less of a corner case here, and as another advantage the
rewriting rule is applied in real time, e.g. you can change the superprojects
rule after the fact and the submodule would automagically make use of it.

>
> One other caveat: I'm not sure if we do insteadOf recursively, but it
> may be surprising to the child "git clone" that we've already applied
> the insteadOf rewriting (especially if the rules are coming from
> ~/.gitconfig and may be applied twice).

When a rule is having effect twice the rule sounds broken. (the outcome
ought to be sufficiently different from the original?)

>
> -Peff

Thanks,
Stefan

Re: url.<base>.insteadOf vs. submodules

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Tue, Feb 21, 2017 at 3:00 PM, Jeff King <peff@peff.net> wrote:
> ...
>> I guess one answer is that this is the wrong approach entirely, and the
>> right one is something like: submodules should understand that they are
>> part of a superproject, and respect some whitelisted set of config from
>> the superproject .git/config file.
>
> This would break one of the core assumptions that submodules
> are "independent" repos.
>
> The way of action is a one way street:
> * The superproject is aware of the submodule and when you invoke a
> command on the superproject, you may mess around with the submodule,
> e.g. update/remove it; absorb its git directory.
> * The submodule is "just" a repository with weird .git link file and a
>   respective core.worktree setup. Currently it doesn't know if it is
>   guided by a superproject.

While that is a good discipline to follow, I think you need to
differenciate the project that is bound as a submodule to a
superproject, and a specific instance of a submodule repository,
i.e. a clone of such a project.

It is true that the Linux kernel project should *NEVER* know your
appliance project only because you happen to use it as a component
of your appliance that happens to use the kernel as one of its
submodules.  But that does not mean your copy of the kernel that
sits in your recursive checkout of your appliance project should
not know anything about your superproject.

This is true even without any submodules.  The Git project itself
does not even care you are Stefan, but you still can and do add
[user] name = "Stefan Beller" to .git/config of your clone of the
Git project.  A clone of the project may want to know more than the
data project itself keeps track of to describe the context in which
the particular clone is being used.  And .git/config is a good place
to keep such pieces of information.

So I would think it is entirely reasonable if "git submodule init
sub" that is run in the superproject to initialize "sub" writes
something in "sub/.git" to tell that "sub" is used in the context of
that particular toplevel superproject and customize its behavour
accordingly.  Perhaps it may want to add the url.*.insteadOf that is
useful for updating the submodule repository when it does "submodule
init", for example.

Re: url.<base>.insteadOf vs. submodules

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> So I would think it is entirely reasonable if "git submodule init
> sub" that is run in the superproject to initialize "sub" writes
> something in "sub/.git" to tell that "sub" is used in the context of
> that particular toplevel superproject and customize its behavour
> accordingly.  Perhaps it may want to add the url.*.insteadOf that is
> useful for updating the submodule repository when it does "submodule
> init", for example.

Of course, "copying" is usually not very desirable, as it invites
one of the copies to go stale.  An actual implementation may just
say "the name of submodule the superproject uses this as is 'foo'".

That way, if such a configuration exists, Git can first do cd-up to
the root of the working tree, go one level up, verify that it is in
a worktree of its superproject, verify that the root of the working
tree it came from was indeed bound to the submodule called 'foo' and
then do the selective/filtered "config-include" Peff outlined.  That
would allow superproject to move submodules around (as opposed to
recording "this submodule is used at this/path of the superproject"
or "the superproject of this submodule is at ../../that/path"), and
does not penalize repositories that are not used as submodules of
any superproject (because the "cd-up, up, verify and include" won't
be done for them).  As opposed to "I am used as a submodule" bit,
recording the name the superproject uses to call the submodule would
also serve as a sanity check measure.

Re: url.<base>.insteadOf vs. submodules

[Stefan Beller]

On Tue, Feb 21, 2017 at 3:37 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> On Tue, Feb 21, 2017 at 3:00 PM, Jeff King <peff@peff.net> wrote:
>> ...
>>> I guess one answer is that this is the wrong approach entirely, and the
>>> right one is something like: submodules should understand that they are
>>> part of a superproject, and respect some whitelisted set of config from
>>> the superproject .git/config file.
>>
>> This would break one of the core assumptions that submodules
>> are "independent" repos.
>>
>> The way of action is a one way street:
>> * The superproject is aware of the submodule and when you invoke a
>> command on the superproject, you may mess around with the submodule,
>> e.g. update/remove it; absorb its git directory.
>> * The submodule is "just" a repository with weird .git link file and a
>>   respective core.worktree setup. Currently it doesn't know if it is
>>   guided by a superproject.
>
> While that is a good discipline to follow, I think you need to
> differenciate the project that is bound as a submodule to a
> superproject, and a specific instance of a submodule repository,
> i.e. a clone of such a project.
>
> It is true that the Linux kernel project should *NEVER* know your
> appliance project only because you happen to use it as a component
> of your appliance that happens to use the kernel as one of its
> submodules.  But that does not mean your copy of the kernel that
> sits in your recursive checkout of your appliance project should
> not know anything about your superproject.

Oh, I see.  For this use case as well as the prompt indicator that
I mentioned in the previous email, the most basic question is
* Do we have a superproject? [yes/no]
The next level of awareness would be
* Where is the superproject? [ <relative path?>]

These questions may not be interesting for a user (they ought to know
about that appliance;) ), but rather for scripted usage, which I think
hints at the lack of a submodule plumbing command.

Currently we only have git-submodule that is a helper used to somehow
cope with submodules. It is used by humans directly and it is listed
under "Main porcelain commands" in our man page.

Probably we'd also do not want to cram this stuff into the already bloated
rev-parse (that has --show-toplevel, which has nothing to do with
parsing revs, but as Jeff put it it is the kitchen sink of Git).

>
> This is true even without any submodules.  The Git project itself
> does not even care you are Stefan, but you still can and do add
> [user] name = "Stefan Beller" to .git/config of your clone of the
> Git project.  A clone of the project may want to know more than the
> data project itself keeps track of to describe the context in which
> the particular clone is being used.  And .git/config is a good place
> to keep such pieces of information.

This analogy is less clear to me than the kernel& appliance.
When applying it to you (user.name=Junio) that has write powers
over the blessed repository, the project cares a lot about you. ;)

> So I would think it is entirely reasonable if "git submodule init
> sub" that is run in the superproject to initialize "sub" writes
> something in "sub/.git" to tell that "sub" is used in the context of
> that particular toplevel superproject and customize its behavour
> accordingly.  Perhaps it may want to add the url.*.insteadOf that is
> useful for updating the submodule repository when it does "submodule
> init", for example.

Do we want to invent a special value for url.*.insteadOf to mean
  "look up in superproject, so I don't have to keep
  a copy that may get stale" ?

Re: url.<base>.insteadOf vs. submodules

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

>> This is true even without any submodules.  The Git project itself
>> does not even care you are Stefan, but you still can and do add
>> [user] name = "Stefan Beller" to .git/config of your clone of the
>> Git project.  A clone of the project may want to know more than the
>> data project itself keeps track of to describe the context in which
>> the particular clone is being used.  And .git/config is a good place
>> to keep such pieces of information.
>
> This analogy is less clear to me than the kernel& appliance.
> When applying it to you (user.name=Junio) that has write powers
> over the blessed repository, the project cares a lot about you. ;)

The name that is recorded in the project history "Stefan Beller"
matters and the project cares about it, when the commit created in
that repository is pulled (or exported and imported via the e-mail
to "git am" route).  But what name you have configured in your
repository's .git/config, or the presense of your particular
repository for that matter, is much less significant (and that
applies to my primary working area as well).  The point is that the
project and a particular clone of it are entities at conceptually
different levels.

>> So I would think it is entirely reasonable if "git submodule init
>> sub" that is run in the superproject to initialize "sub" writes
>> something in "sub/.git" to tell that "sub" is used in the context of
>> that particular toplevel superproject and customize its behavour
>> accordingly.  Perhaps it may want to add the url.*.insteadOf that is
>> useful for updating the submodule repository when it does "submodule
>> init", for example.
>
> Do we want to invent a special value for url.*.insteadOf to mean
>   "look up in superproject, so I don't have to keep
>   a copy that may get stale" ?

My gut feeling is that we should do the selective/filtered include
Peff mentioned when a repository is known to be used as a submodule
of somebody else.

Re: url.<base>.insteadOf vs. submodules

[Jon Loeliger]

So, like, Junio C Hamano said:
> Stefan Beller <sbeller@google.com> writes:
> 
> > Do we want to invent a special value for url.*.insteadOf to mean
> >   "look up in superproject, so I don't have to keep
> >   a copy that may get stale" ?
> 
> My gut feeling is that we should do the selective/filtered include
> Peff mentioned when a repository is known to be used as a submodule
> of somebody else.

Does the management of these submodue-related config values
become easier if, instead of placing them in .config, we
place them in a git/.context file?

jdl

Re: url.<base>.insteadOf vs. submodules

[Junio C Hamano]

Jon Loeliger <jdl@jdl.com> writes:

> So, like, Junio C Hamano said:
>> Stefan Beller <sbeller@google.com> writes:
>> 
>> > Do we want to invent a special value for url.*.insteadOf to mean
>> >   "look up in superproject, so I don't have to keep
>> >   a copy that may get stale" ?
>> 
>> My gut feeling is that we should do the selective/filtered include
>> Peff mentioned when a repository is known to be used as a submodule
>> of somebody else.
>
> Does the management of these submodue-related config values
> become easier if, instead of placing them in .config, we
> place them in a git/.context file?

Do you mean that Git users that use submodules adopt a convention
where a separate file in $GIT_DIR of the toplevel superproject holds
pieces of configuration that are meant to be shared between the
superproject and across all its submodules, and the $GIT_DIR/config
file in submodules and the superproject all include that shared one
via include.path mechanism?

That may allow us to do without being responsible for sifting of
configuration variables into safe and unsafe bins.

I dunno.

Re: url.<base>.insteadOf vs. submodules

[Jeff King]

On Wed, Feb 22, 2017 at 09:36:12AM -0800, Junio C Hamano wrote:

> >> My gut feeling is that we should do the selective/filtered include
> >> Peff mentioned when a repository is known to be used as a submodule
> >> of somebody else.
> >
> > Does the management of these submodue-related config values
> > become easier if, instead of placing them in .config, we
> > place them in a git/.context file?
> 
> Do you mean that Git users that use submodules adopt a convention
> where a separate file in $GIT_DIR of the toplevel superproject holds
> pieces of configuration that are meant to be shared between the
> superproject and across all its submodules, and the $GIT_DIR/config
> file in submodules and the superproject all include that shared one
> via include.path mechanism?
> 
> That may allow us to do without being responsible for sifting of
> configuration variables into safe and unsafe bins.
> 
> I dunno.

Hmm. I certainly like that we punt on having to decide on the "should
this be shared with submodules" decision. That makes the end result more
flexible, and we don't have to get into a never-ending stream of
"whitelist this config option" patches.

My only concern is that it's not as discoverable. In the situation that
kicked off this thread, somebody put url.X.insteadOf into their
super-project .git/config, expecting it to work in the submodules. That
_still_ wouldn't work with this proposal. They'd have to:

  1. Put it in .git/context (or whatever we call it)

  2. Maybe add include.path=context in .git/config if they want the
     config shared with the super-project (or this could be automatic?)

I guess it gives _a_ solution, which is more than we have now, but it
doesn't feel very ergonomic.

-Peff

Re: url.<base>.insteadOf vs. submodules

[Stefan Beller]

On Wed, Feb 22, 2017 at 10:57 AM, Jeff King <peff@peff.net> wrote:
> On Wed, Feb 22, 2017 at 09:36:12AM -0800, Junio C Hamano wrote:
>
>> >> My gut feeling is that we should do the selective/filtered include
>> >> Peff mentioned when a repository is known to be used as a submodule
>> >> of somebody else.
>> >
>> > Does the management of these submodue-related config values
>> > become easier if, instead of placing them in .config, we
>> > place them in a git/.context file?
>>
>> Do you mean that Git users that use submodules adopt a convention
>> where a separate file in $GIT_DIR of the toplevel superproject holds
>> pieces of configuration that are meant to be shared between the
>> superproject and across all its submodules, and the $GIT_DIR/config
>> file in submodules and the superproject all include that shared one
>> via include.path mechanism?
>>
>> That may allow us to do without being responsible for sifting of
>> configuration variables into safe and unsafe bins.
>>
>> I dunno.
>
> Hmm. I certainly like that we punt on having to decide on the "should
> this be shared with submodules" decision. That makes the end result more
> flexible, and we don't have to get into a never-ending stream of
> "whitelist this config option" patches.
>
> My only concern is that it's not as discoverable. In the situation that
> kicked off this thread, somebody put url.X.insteadOf into their
> super-project .git/config, expecting it to work in the submodules. That
> _still_ wouldn't work with this proposal. They'd have to:
>
>   1. Put it in .git/context (or whatever we call it)
>
>   2. Maybe add include.path=context in .git/config if they want the
>      config shared with the super-project (or this could be automatic?)
>
> I guess it gives _a_ solution, which is more than we have now, but it
> doesn't feel very ergonomic.

Well, currently ".git/config" is the one and only blessed way to configure
a single repo and our documentation and user expectations reflect that.
Once git-worktree takes off (which has per working tree configuration files)
it doesn't feel as obscure anymore to have multiple config files.

The working trees will share the $GIT_COMMON_DIR/config file for
all working trees and have its own config file at $GIT_DIR/config.worktree
in its respective git directories. C.f.
https://public-inbox.org/git/20170110112524.12870-2-pclouds@gmail.com/

So I could imagine that we just introduce another config file
config.submodules which is source'd by the submodules.
Then the hard part becomes to decide which config value to put
in which config file. (We'd still be left to guess where to put some initial
new configuration value. config or config.submodules. Any update of a
value can just stay in its respective file. And I don't think we'd want
to invent a config option that tells us which policy we use where to
put config options. That sounds just scary.)

Re: url.<base>.insteadOf vs. submodules

[Jeff King]

On Tue, Feb 21, 2017 at 03:16:27PM -0800, Stefan Beller wrote:

> > I guess one answer is that this is the wrong approach entirely, and the
> > right one is something like: submodules should understand that they are
> > part of a superproject, and respect some whitelisted set of config from
> > the superproject .git/config file.
> 
> This would break one of the core assumptions that submodules
> are "independent" repos.

Yeah, that was the "first half" that I said was hard. :)

You could rationalize it under the fact that they _are_ independent
repos; we're just adding a new config source.  Arguably it could be a
feature for any repository embedded inside the working tree of another,
submodule or not, to consider the outer repository as a (limited) source
of config.

But there are probably a lot of irritating corner cases with the whole
concept unless we apply a strict whitelist of keys (e.g., you probably
don't want remote.* to be propagated). And as the recent
GIT_CONFIG_PARAMETERS whitelist showed, that approach ended up confusing
and annoying.

So maybe the whole thing is insane, and the right answer is that config
values should go into ~/.gitconfig. And we may need better tools there
for limiting that global config to certain parts of the tree (like Duy's
conditional include thing).

> Though I do not know if this is actually a good assumption.
> e.g. "[PATCH v2] git-prompt.sh: add submodule indicator"
> https://public-inbox.org/git/1486075892-20676-2-git-send-email-email@benjaminfuchs.de/
> really had trouble in the first version to nail down how to tell you are in
> a submodule, but people want to know that.

Right, I think it's an interesting thing to know, but I agree there are
probably a lot of corner cases.

> Maybe we need to change that fundamental assumption.
> So a more sophisticated way (thinking long term here) would be
> to include the superprojects config file (with exceptions), and that
> config file has more priority than e.g. the ~/.gitconfig file, but less
> than the submodules own $GIT_DIR/config file.

Yeah, that priority matches what I had been thinking.

> > One other caveat: I'm not sure if we do insteadOf recursively, but it
> > may be surprising to the child "git clone" that we've already applied
> > the insteadOf rewriting (especially if the rules are coming from
> > ~/.gitconfig and may be applied twice).
> 
> When a rule is having effect twice the rule sounds broken. (the outcome
> ought to be sufficiently different from the original?)

If you have:

  url.bar.insteadOf=foo
  url.baz.insteadOf=bar

do we convert "foo" to "baz"? If so, then I think applying the rules
again shouldn't matter. But if we don't, and only do a single level,
then having the caller rewrite the URL before it hands it to "git clone"
means we may end up unexpectedly doing two levels of rewriting.

-Peff

Re: url.<base>.insteadOf vs. submodules

[Stefan Beller]

On Tue, Feb 21, 2017 at 3:40 PM, Jeff King <peff@peff.net> wrote:

>> > One other caveat: I'm not sure if we do insteadOf recursively, but it
>> > may be surprising to the child "git clone" that we've already applied
>> > the insteadOf rewriting (especially if the rules are coming from
>> > ~/.gitconfig and may be applied twice).
>>
>> When a rule is having effect twice the rule sounds broken. (the outcome
>> ought to be sufficiently different from the original?)
>
> If you have:
>
>   url.bar.insteadOf=foo
>   url.baz.insteadOf=bar
>
> do we convert "foo" to "baz"? If so, then I think applying the rules
> again shouldn't matter. But if we don't, and only do a single level,
> then having the caller rewrite the URL before it hands it to "git clone"
> means we may end up unexpectedly doing two levels of rewriting.
>

I see. Thanks for the example. So really what we want is to record the
unencumbered URL (with no rewriting) and then at run time lookup various
places of url.*.insteadOf (which might change with the git version
that you use)

Thanks,
Stefan