Why are submodules not automatically handled by default or at least configurable to do so?

Why are submodules not automatically handled by default or at least configurable to do so?

[John Smith]

I found that I use submodules much, much more often in my git projects than I used externals
in Subversion and the reason is that git encourages/forces to organize large projects into
smaller repositories, one reason for this being that subversion allows to check out parts of 
a repository while git does not.

But when I clone a git repository with subprojects, I (and everyone else) has to remember to
add the --recursive option. When switching between branches with different versions/commits of the 
submodules everyone has to remember to update the submodules. When updating a submodule 
everyone has to remember to recurse there too. 

Basically, everything with submodules has to be done manually every time and there seems 
to be no way to change that default.

Why is that? Basically all the time I use submodules I would want automatic handling of 
submodules to happen and I cannot  remember having had a single situation where I would 
not have wanted it to happen. So  why does git default to doing nothing? 
Why does it not provide a way to enable automatic
pulling/updating of submodules e.g. when cloning or switching branches?
When would people routinely check out a branch and want to stay with the submodules as 
the have been checked out for the old branch?

I honestly do not understand it. 

John

____________________________________________________________
Can't remember your password? Do you need a strong and secure password?
Use Password manager! It stores your passwords & protects your account.
Check it out at http://mysecurelogon.com/manager

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Chris Packham]

On Mon, Oct 26, 2015 at 12:10 PM, John Smith <johsmi9933@inbox.com> wrote:
> I found that I use submodules much, much more often in my git projects than I used externals
> in Subversion and the reason is that git encourages/forces to organize large projects into
> smaller repositories, one reason for this being that subversion allows to check out parts of
> a repository while git does not.
>
> But when I clone a git repository with subprojects, I (and everyone else) has to remember to
> add the --recursive option. When switching between branches with different versions/commits of the
> submodules everyone has to remember to update the submodules. When updating a submodule
> everyone has to remember to recurse there too.

The config option fetch.recurseSubmodules exists. It's not quite the
same as what git clone --recurse-submodules does but it's a start.

>
> Basically, everything with submodules has to be done manually every time and there seems
> to be no way to change that default.
>
> Why is that? Basically all the time I use submodules I would want automatic handling of
> submodules to happen and I cannot  remember having had a single situation where I would
> not have wanted it to happen. So  why does git default to doing nothing?

It's hard to pick a default that suits every workflow that submodules
support. Also with submodules there is a chicken-and-egg scenario.
While you can put things in ~/.gitconfig most of what you'd want to
configure when using submodules would be in super/.git/config but that
doesn't exist until you've cloned super.git.

> Why does it not provide a way to enable automatic
> pulling/updating of submodules e.g. when cloning or switching branches?

I believe Jens and Stefan (Cc'd) have been doing some great work in
this area. Jens even posted his todo list a few days ago
(https://github.com/jlehmann/git-submod-enhancements/wiki).

> When would people routinely check out a branch and want to stay with the submodules as
> the have been checked out for the old branch?
>
> I honestly do not understand it.
>
> John
>
> ____________________________________________________________
> Can't remember your password? Do you need a strong and secure password?
> Use Password manager! It stores your passwords & protects your account.
> Check it out at http://mysecurelogon.com/manager
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Stefan Beller]

On Sun, Oct 25, 2015 at 5:56 PM, Chris Packham <judge.packham@gmail.com> wrote:
> On Mon, Oct 26, 2015 at 12:10 PM, John Smith <johsmi9933@inbox.com> wrote:
>> I found that I use submodules much, much more often in my git projects than I used externals
>> in Subversion and the reason is that git encourages/forces to organize large projects into
>> smaller repositories, one reason for this being that subversion allows to check out parts of
>> a repository while git does not.
>>
>> But when I clone a git repository with subprojects, I (and everyone else) has to remember to
>> add the --recursive option. When switching between branches with different versions/commits of the
>> submodules everyone has to remember to update the submodules. When updating a submodule
>> everyone has to remember to recurse there too.
>
> The config option fetch.recurseSubmodules exists. It's not quite the
> same as what git clone --recurse-submodules does but it's a start.
>
>>
>> Basically, everything with submodules has to be done manually every time and there seems
>> to be no way to change that default.
>>
>> Why is that? Basically all the time I use submodules I would want automatic handling of
>> submodules to happen and I cannot  remember having had a single situation where I would
>> not have wanted it to happen. So  why does git default to doing nothing?

IIUC at the time submodules were invented, there was need for lots of
code to be written.
Each command needed new code to deal with submodules. As there was not
enough people/time
to do it properly, the "do nothing" was the safest action which could
be added fast.

>
> It's hard to pick a default that suits every workflow that submodules
> support. Also with submodules there is a chicken-and-egg scenario.
> While you can put things in ~/.gitconfig most of what you'd want to
> configure when using submodules would be in super/.git/config but that
> doesn't exist until you've cloned super.git.
>
>> Why does it not provide a way to enable automatic
>> pulling/updating of submodules e.g. when cloning or switching branches?
>
> I believe Jens and Stefan (Cc'd) have been doing some great work in
> this area. Jens even posted his todo list a few days ago
> (https://github.com/jlehmann/git-submod-enhancements/wiki).

Yeah I would also point at Jens' wiki today.

All I did up to now was rewriting parts of the submodule code in C
(git submodule update specifically), while the code/patches you find at Jens'
copy of Git includes already lots of useful stuff such as `git
checkout --recurse-submodules`
(IIRC you don't need to type --recurse-submodules if you configured
that to be the default)

>
>> When would people routinely check out a branch and want to stay with the submodules as
>> the have been checked out for the old branch?

As said above, it was a sane choice which could be implemented fast, IIUC.

I mean what would happen if you had commits made in the submodule, or
just a dirty working tree?

>>
>> I honestly do not understand it.
>>
>> John
>>

Stefan

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> IIUC at the time submodules were invented, there was need for lots of
> code to be written.
> Each command needed new code to deal with submodules. As there was not
> enough people/time
> to do it properly, the "do nothing" was the safest action which could
> be added fast.

That is quite different from how I remember.  Soon after Linus and I
added the Gitlink in early Apr 20007, an early subproject/gitlink
(thought) experiment was started with help from folks like Steven
Grimm, Jan Hudec, Petr Baudis, Alex Riesen etc.  The first principle
of the design throughout that era was "we admit that we do not know
all the use cases, so let's start small and solid and make sure that
small-and-solid thing can later be enhanced as people discover the
way how they want to work" (you can see me expressing that sentiment
in $gmane/48287, for example).

So it wasn't "not enough people to do it properly" at all.  It was
"we admit that we do not know what is proper, so we defer to actual
users to define what is proper for them".

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Nazri Ramliy]

On Mon, Oct 26, 2015 at 7:10 AM, John Smith <johsmi9933@inbox.com> wrote:
> When would people routinely check out a branch and want to stay with the submodules as
> the have been checked out for the old branch?

I do this a lot. At my $dayjob we have a super project with bunch of
sub projects.
Each subproject has its corresponding rpm spec file in the
superproject - it's quite
often that I work on a "git-merge-base--octopus" branch that updates only the
spec files and nothing else - so when changing between branches I
don't care what
states the submodules are in. When the fixes to the spec files are ready I just
checkout to the respective branches and merge in the changes - I don't actively
do "git submodule update" when switching to different branches.

nazri

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Jens Lehmann]

Am 26.10.2015 um 05:48 schrieb Nazri Ramliy:
> On Mon, Oct 26, 2015 at 7:10 AM, John Smith <johsmi9933@inbox.com> wrote:
>> When would people routinely check out a branch and want to stay with the submodules as
>> the have been checked out for the old branch?
>
> I do this a lot. At my $dayjob we have a super project with bunch of
> sub projects.
> Each subproject has its corresponding rpm spec file in the
> superproject - it's quite
> often that I work on a "git-merge-base--octopus" branch that updates only the
> spec files and nothing else - so when changing between branches I
> don't care what
> states the submodules are in. When the fixes to the spec files are ready I just
> checkout to the respective branches and merge in the changes - I don't actively
> do "git submodule update" when switching to different branches.

Which seems a bit error prone as you could forget to update the submodules
and build incorrect rpms from them, or am I missing something?

I understand why you don't need to update the submodules every time, but
would it hurt your workflow if they did (but don't get me wrong, that will
always be configurable).

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Nazri Ramliy]

On Tue, Oct 27, 2015 at 12:56 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Which seems a bit error prone as you could forget to update the submodules
> and build incorrect rpms from them, or am I missing something?

For my case I'm not building the rpms directly after merging in the fixes
done in the octopus branch, so I don't care much about the state of the
submodules after the merge since I know that the octopus branch does
not modify any submodules. The rpms are built later on, possibly on
another machine where the submodules are updated w.r.t to each branch
in a release commit.

> I understand why you don't need to update the submodules every time, but
> would it hurt your workflow if they did (but don't get me wrong, that will
> always be configurable).

I'd say it depends - for the times when all I want to do is work on plain files
in the superproject (on an octopus branch for example) I don't want git to
automatically update the submodules everytime I switch branches - it would
hurt my productivity as there will be more disk activities due to files being
checked out unnecessarily for updating the submodules.

The submodule states are not committed that often into the superproject,
they are done normally when we're cutting a new release. During day-to-day
development each developer runs a script that pulls in the latest commits
for "hot" submodules.

git not updating the submodule state automatically is actually a convenient
for my particular use case here.

nazri

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Nick]

I too am interested in finding ways to automate working with submodules, 
as it's a particular pain point with my colleagues.  They frequently 
shoot themselves in the foot trying to branch and merge a project with 
submodules, resulting in a broken build and grumpy comments about git 
(or possibly about me, as the "Git Advocate").  And they're right, it is 
awkward.

Whether or not any sensible default configuration exists, users do need 
a way to avoid excessively complicated workflows, or people are just 
going to avoid using submodules, or perhaps Git. Hand-rolled solutions 
might be better than nothing, but I would expect that this is a common 
issue which would benefit from a built-in solution.  For instance, so 
that one can branch and merge the whole project without duplicating the 
work for each submodule.

Am I correct in thinking there isn't anything which does these kind of 
things yet?

Thanks!

Nick

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Davide Fiorentino]

Why not set alias(es) for that?

Best,
Davide
> On 27 Oct 2015, at 10:50, Nick <oinksocket@letterboxes.org> wrote:
> 
> I too am interested in finding ways to automate working with submodules, as it's a particular pain point with my colleagues.  They frequently shoot themselves in the foot trying to branch and merge a project with submodules, resulting in a broken build and grumpy comments about git (or possibly about me, as the "Git Advocate").  And they're right, it is awkward.
> 
> Whether or not any sensible default configuration exists, users do need a way to avoid excessively complicated workflows, or people are just going to avoid using submodules, or perhaps Git. Hand-rolled solutions might be better than nothing, but I would expect that this is a common issue which would benefit from a built-in solution.  For instance, so that one can branch and merge the whole project without duplicating the work for each submodule.
> 
> Am I correct in thinking there isn't anything which does these kind of things yet?
> 
> Thanks!
> 
> Nick
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Nick]

On 27/10/15 10:56, Davide Fiorentino wrote:
> Why not set alias(es) for that?

That counts as a hand-rolled (i.e. ad-hoc) solution.  So not out of the 
question, but I'd rather point my colleagues at something tried and 
tested, rather than simply re-invent wheels, possibly badly.

I'd be interested if there are some out there I could adopt?

But oh yes, there is another difficulty with aliases.  Eclipse users on 
Windows: they don't tend to love it if you tell them to install Cygwin, 
open a shell and type things into it.  The Eclipse experience is 
somewhat fraught already (e.g. EGit running out of memory cloning 
moderate-sized repos, and other UI difficulties). I'd be surprised if 
defining aliases is going to help their user experience.  I'm 
considering suggesting they switch to to IntelliJ, but that's also 
asking quite a lot of people who may be reluctant to relearn their whole 
workflow again, and I'd need to do the research to ensure this doesn't 
just make things more confusing.

Cheers,

N

Re: Why are submodules not automatically handled by default or at least configurable to do so?

[Konstantin Khomoutov]

On Tue, 27 Oct 2015 11:40:15 +0000
Nick <oinksocket@letterboxes.org> wrote:

> > Why not set alias(es) for that?
[...]
> But oh yes, there is another difficulty with aliases.  Eclipse users
> on Windows:
[...]

Not to counter your actual argument, but AFAIK EGit uses JGit which is
a Java implementation which does not call out to the "real" Git binary.
This basically means that if something gets implemented in the stock
Git, this won't affect JGit and EGit untill their respective
maintainers implement the same feature.

> they don't tend to love it if you tell them to install
> Cygwin, open a shell and type things into it.

On Windows, you typically want them to use Git for Windows, not
Cygwin.  Various GUI front-ends to Git working on Windows (such as Git
Extentions and TortoiseGit) rely on GfW to work as well.