Subtree with submodule inside?

Subtree with submodule inside?

[Robert Dailey]

Is this even possible? The .gitmodule file has to be at the root of
the repository, AFAIK. So if the subtree is inherently not at the
root, how does it manage its own submodules?

Basically I have a common library that also keeps a submodule of third
party dependencies (binaries). Each super project that depends on this
common library (each super project has its own repository) will add it
as a subtree. So what happens to the submodule with this setup?

Thanks.

Re: Subtree with submodule inside?

[Junio C Hamano]

Robert Dailey <rcdailey.lists@gmail.com> writes:

> Is this even possible? The .gitmodule file has to be at the root of
> the repository, AFAIK. So if the subtree is inherently not at the
> root, how does it manage its own submodules?
>
> Basically I have a common library that also keeps a submodule of third
> party dependencies (binaries). Each super project that depends on this
> common library (each super project has its own repository) will add it
> as a subtree. So what happens to the submodule with this setup?

My knee-jerk reaction would be "subtree would break submodules
badly, don't use it" ;-).

After all, I invented subtree merge as an ugly interim workaround
before submodule subsystem got into a usable shape, hoping that new
projects can use submodules without resorting to subtree merges.

Re: Subtree with submodule inside?

[Robert Dailey]

On Wed, Aug 6, 2014 at 12:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
> My knee-jerk reaction would be "subtree would break submodules
> badly, don't use it" ;-).
>
> After all, I invented subtree merge as an ugly interim workaround
> before submodule subsystem got into a usable shape, hoping that new
> projects can use submodules without resorting to subtree merges.

(Sorry Junio, I forgot to "Reply all" and accidentally sent this only
to you. Resending to group)

Hi Junio. I agree it certainly does sound evil and defeats the purpose
of the goal of ultimately creating a simple workflow :)

Could you go over how you feel the submodule system got into "usable"
shape? It still seems to still have all of the problems I still hate:

* If you branch a production branch to implement a feature that also
impacts the submodule, you need to branch the submodule. There is no
easy interface for this and rather clumbsy, especially if people are
using GUI tools like SourceTree (most of the people on my team are).
* Pull requests do not include submodule changes in the review (we use
Atlassian Stash). So 1 pull request per submodule has to be performed
and they have to be merged in the appropriate order.
* Submodules complicate an otherwise beautiful workflow that Git is
supposed to provide (branching) in many ways, and also there are
restrictions on the order in which you modify parent repository vs
submodule.

If I'm going to put my common code in a separate module for the
purposes of splitting apart projects that depend on the common code, I
want the workflows that Git provides to remain simple. Subtrees seem
to do that, you just sync with the common repo every now and then.

Re: Subtree with submodule inside?

[Jens Lehmann]

Am 06.08.2014 um 20:18 schrieb Robert Dailey:
> On Wed, Aug 6, 2014 at 12:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> My knee-jerk reaction would be "subtree would break submodules
>> badly, don't use it" ;-).
>>
>> After all, I invented subtree merge as an ugly interim workaround
>> before submodule subsystem got into a usable shape, hoping that new
>> projects can use submodules without resorting to subtree merges.
> 
> (Sorry Junio, I forgot to "Reply all" and accidentally sent this only
> to you. Resending to group)
> 
> Hi Junio. I agree it certainly does sound evil and defeats the purpose
> of the goal of ultimately creating a simple workflow :)
> 
> Could you go over how you feel the submodule system got into "usable"
> shape? It still seems to still have all of the problems I still hate:
> 
> * If you branch a production branch to implement a feature that also
> impacts the submodule, you need to branch the submodule. There is no
> easy interface for this and rather clumbsy, especially if people are
> using GUI tools like SourceTree (most of the people on my team are).

There were thoughts about having "git branch" optionally create a
branch in the submodule too. But in a lot of real world scenarios
that won't help because the same branch name won't necessarily make
sense in superproject and submodule at the same time, so you'd have
to create different branches anyway. But if it makes a significant
amount of users happy, I think adding --recurse-submodules to "git
branch" might make sense to make life easier for them.

I do not know SourceTree myself, but do you have any ideas you'd
like to share how you could envision to make branching less tedious?
I hope it at least shows you that there are modifications inside the
submodule while examining the superproject and provides an easy way
to start a new instance inside the submodule to commit the changes
there? Otherwise it'd make life unnecessarily hard for you.

> * Pull requests do not include submodule changes in the review (we use
> Atlassian Stash). So 1 pull request per submodule has to be performed
> and they have to be merged in the appropriate order.

Which seems to be a chore in your scenario, but is just the Right
Thing to do when someone else maintains the submodule. Given the
danger of a rebase in the submodule pulling away the commit recorded
in the superproject it is considered best practice to always merge
in the submodule into a never-to-be-rebased branch first before
committing the superproject. But I agree it is twice the work.

> * Submodules complicate an otherwise beautiful workflow that Git is
> supposed to provide (branching) in many ways, and also there are
> restrictions on the order in which you modify parent repository vs
> submodule.

I cannot remember anyone coming up with a better workflow when
having distinct repos (superproject and submodule). Subtree on the
other hand has other issues (like you can easily forget to push a
fix to a subtree to its upstream, which is something submodules
make you aware of). And the submodule workflow is as beautiful as
ever if you consider superproject and submodule different repos
(which they are), and then it is obvious that you have to do
everything twice ;-)

But I do have the impression that a lot of third party tools are
still rather submodule ignorant and of not much help there.

> If I'm going to put my common code in a separate module for the
> purposes of splitting apart projects that depend on the common code, I
> want the workflows that Git provides to remain simple. Subtrees seem
> to do that, you just sync with the common repo every now and then.

If you are only interested in single "from upstream into my repo"
workflow and don't care much about pushing your local changes back
to the common code repo, subtrees are easier to use than submodules.
But if you need or want to contribute your changes back, I believe
submodules gain the upper hand rather quickly.

But back to your original question: I do no see a way to import
another repo using a submodule via subtree either. You'd have to
consistently use submodules to make that work (even if that means
that you'll have three repos to think about, one being embedded in
the other).

Re: Subtree with submodule inside?

[Jonathan Nieder]

Jens Lehmann wrote:

> There were thoughts about having "git branch" optionally create a
> branch in the submodule too. But in a lot of real world scenarios
> that won't help because the same branch name won't necessarily make
> sense in superproject and submodule at the same time

So, here is how I think git could behave to make that into a
non-issue:

 1. Commands like 'git checkout' able to recurse into submodules, so
    when you switch branches in the superproject, the files in the
    submodule are at the right commit, too.

    Luckily your series does that.  Ronnie was helping me with thoughts
    about how to simplify the patch a little, and I'd be happy to talk
    with or coordinate with anyone else interested (by email or on IRC
    --- I am jrnieder on Freenode).

 2. Submodules aware of their superproject and of the parent's branches.
    In other words, submodules would act as thought under refs/ they
    had a symlink

	parent -> ../../../refs

    So you could do

	git checkout --recurse-submodules master

	cd path/to/submodule
	git checkout parent/heads/next

    This would avoid danger from "git gc" in submodules and would
    get rid of most of the motivation for named branches in the
    submodule, I'd think.

 3. That's it.

Sensible?

Thanks,
Jonathan

Re: Subtree with submodule inside?

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

>  2. Submodules aware of their superproject and of the parent's branches.
>     In other words, submodules would act as thought under refs/ they
>     had a symlink
>
> 	parent -> ../../../refs
>
>     So you could do
>
> 	git checkout --recurse-submodules master
>
> 	cd path/to/submodule
> 	git checkout parent/heads/next
>
>     This would avoid danger from "git gc" in submodules and would
>     get rid of most of the motivation for named branches in the
>     submodule, I'd think.

Are you assuming that they share their object stores?

Re: Subtree with submodule inside?

[Jonathan Nieder]

Junio C Hamano wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>>  2. Submodules aware of their superproject and of the parent's branches.
>>     In other words, submodules would act as though under refs/ they
>>     had a symlink
>>
>> 	parent -> ../../../refs
>>
>>     So you could do
>>
>> 	git checkout --recurse-submodules master
>>
>> 	cd path/to/submodule
>> 	git checkout parent/heads/next
>>
>>     This would avoid danger from "git gc" in submodules and would
>>     get rid of most of the motivation for named branches in the
>>     submodule, I'd think.
>
> Are you assuming that they share their object stores?

No.  The 'symlink' thing is a think-o.  (When trying to explain the
idea I ended up oversimplifying and speaking nonsense.)

What I wanted to say is that parent/heads/next would be a way to
refer from the submodule to the same commit as

	refs/heads/next:path/to/submodule

refers to in the parent.

Thanks for clarifying,
Jonathan

Re: Subtree with submodule inside?

[Heiko Voigt]

On Wed, Aug 06, 2014 at 04:51:52PM -0700, Jonathan Nieder wrote:
> Junio C Hamano wrote:
> > Jonathan Nieder <jrnieder@gmail.com> writes:
> 
> >>  2. Submodules aware of their superproject and of the parent's branches.
> >>     In other words, submodules would act as though under refs/ they
> >>     had a symlink
> >>
> >> 	parent -> ../../../refs
> >>
> >>     So you could do
> >>
> >> 	git checkout --recurse-submodules master
> >>
> >> 	cd path/to/submodule
> >> 	git checkout parent/heads/next
> >>
> >>     This would avoid danger from "git gc" in submodules and would
> >>     get rid of most of the motivation for named branches in the
> >>     submodule, I'd think.
> >
> > Are you assuming that they share their object stores?
> 
> No.  The 'symlink' thing is a think-o.  (When trying to explain the
> idea I ended up oversimplifying and speaking nonsense.)
> 
> What I wanted to say is that parent/heads/next would be a way to
> refer from the submodule to the same commit as
> 
> 	refs/heads/next:path/to/submodule
> 
> refers to in the parent.

I like this idea. It could solve many issues and help in many cases I think.
Since we are currently quite busy with other things I took the liberty of
adding an ideas section in Jens submodule wiki[1]. This way we do not forget
about it and/or can refer others to it more easily.

I would appreciate if someone could have a look whether I described the idea
clearly enough.

Cheers Heiko

[1] https://github.com/jlehmann/git-submod-enhancements/wiki#dynamic-superproject-refs-in-submodules