Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

[Phil Hord]

On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
>>
>> Some devs of my Team complained that with submodules it is
>> difficult to see the “full picture” of the difference
>> between two SHA1 on the root project, as the submodules
>> would just show as different SHA1s. When you Google
>> “subtree submodules” you find other opinions as well:
>>
>> Just to mention a few:
>> -
>> https://codingkilledthecat.wordpress.com/2012/04/28/why-y
>> our-company-shouldnt-use-git-submodules/ -
>> http://blogs.atlassian.com/2013/05/alternatives-to-git-su
>> bmodule-git-subtree/
>>
>> To be honest with you, I am absolutely fine with
>> submodules as I can easily leave with the “extra pain” of
>> diffing by hand recursively on submodules. But it is true
>> that it may happen to either forget to do a git submodule
>> update or otherwise forget you are in a detached branch
>> and start committing “on the air” without a branch.

...

> Ideally, as a "git clone --recursive" already exists, I would like to
> see a "git diff --recursive" that goes through the submodules as well :-)
>
> Something possibly to propose to the Git mailing list?


I've worked on git diff --recursive a bit myself, along with some
simpler use cases (git ls-tree --recursive) as POCs. I think some of
the needs there begin to have ui implications which could be
high-friction. I really want to finish it someday, but I've been too
busy lately at $job, and now my experiments are all rather stale.

It would be a good discussion to have over at the git list (copied).
Heiko and Jens have laid some new groundwork in this area and it may
be a good time to revisit it.  Or maybe they've even moved deeper than
that; I have been distracted for well over a year now.

Phil

-- 
-- 
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

[Luca Milanesio]

Thank you Phil, you anticipated me :-)

Luca.

> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
> 
> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
>>> 
>>> Some devs of my Team complained that with submodules it is
>>> difficult to see the “full picture” of the difference
>>> between two SHA1 on the root project, as the submodules
>>> would just show as different SHA1s. When you Google
>>> “subtree submodules” you find other opinions as well:
>>> 
>>> Just to mention a few:
>>> -
>>> https://codingkilledthecat.wordpress.com/2012/04/28/why-y
>>> our-company-shouldnt-use-git-submodules/ -
>>> http://blogs.atlassian.com/2013/05/alternatives-to-git-su
>>> bmodule-git-subtree/
>>> 
>>> To be honest with you, I am absolutely fine with
>>> submodules as I can easily leave with the “extra pain” of
>>> diffing by hand recursively on submodules. But it is true
>>> that it may happen to either forget to do a git submodule
>>> update or otherwise forget you are in a detached branch
>>> and start committing “on the air” without a branch.
> 
> ...
> 
>> Ideally, as a "git clone --recursive" already exists, I would like to
>> see a "git diff --recursive" that goes through the submodules as well :-)
>> 
>> Something possibly to propose to the Git mailing list?
> 
> 
> I've worked on git diff --recursive a bit myself, along with some
> simpler use cases (git ls-tree --recursive) as POCs. I think some of
> the needs there begin to have ui implications which could be
> high-friction. I really want to finish it someday, but I've been too
> busy lately at $job, and now my experiments are all rather stale.
> 
> It would be a good discussion to have over at the git list (copied).
> Heiko and Jens have laid some new groundwork in this area and it may
> be a good time to revisit it.  Or maybe they've even moved deeper than
> that; I have been distracted for well over a year now.
> 
> Phil

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

[Stefan Beller]

On 06.06.2015 12:53, Luca Milanesio wrote:
> Thank you Phil, you anticipated me :-)
> 
> Luca.
> 
>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>
>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
>>>>
>>>> Some devs of my Team complained that with submodules it is
>>>> difficult to see the “full picture” of the difference
>>>> between two SHA1 on the root project, as the submodules
>>>> would just show as different SHA1s. When you Google
>>>> “subtree submodules” you find other opinions as well:
>>>>
>>>> Just to mention a few:
>>>> -
>>>> https://codingkilledthecat.wordpress.com/2012/04/28/why-y
>>>> our-company-shouldnt-use-git-submodules/ -
>>>> http://blogs.atlassian.com/2013/05/alternatives-to-git-su
>>>> bmodule-git-subtree/
>>>>
>>>> To be honest with you, I am absolutely fine with
>>>> submodules as I can easily leave with the “extra pain” of
>>>> diffing by hand recursively on submodules. But it is true
>>>> that it may happen to either forget to do a git submodule
>>>> update or otherwise forget you are in a detached branch
>>>> and start committing “on the air” without a branch.
>>
>> ...
>>
>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>> see a "git diff --recursive" that goes through the submodules as well :-)
>>>
>>> Something possibly to propose to the Git mailing list?
>>
>>
>> I've worked on git diff --recursive a bit myself, along with some
>> simpler use cases (git ls-tree --recursive) as POCs. I think some of
>> the needs there begin to have ui implications which could be
>> high-friction. I really want to finish it someday, but I've been too
>> busy lately at $job, and now my experiments are all rather stale.
>>
>> It would be a good discussion to have over at the git list (copied).
>> Heiko and Jens have laid some new groundwork in this area and it may
>> be a good time to revisit it.  Or maybe they've even moved deeper than
>> that; I have been distracted for well over a year now.
>>

Glad you're working (or planning to) working on submodulues. This is
also on my todo list for the next months as well.

I'd review stuff in that area if you're looking for reviewers.

Stefan

>> Phil
> 
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

[Jens Lehmann]

Am 07.06.2015 um 08:26 schrieb Stefan Beller:
> On 06.06.2015 12:53, Luca Milanesio wrote:
>>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
>>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>>> see a "git diff --recursive" that goes through the submodules as well :-)
>>>>
>>>> Something possibly to propose to the Git mailing list?

Such an option makes lots of sense to me (though "--recurse-submodules"
should be its name for consistency reasons). This could be an alias for
"--submodule=full", as the "--submodule" option controls the format of
submodule diffs.

>>> I've worked on git diff --recursive a bit myself, along with some
>>> simpler use cases (git ls-tree --recursive) as POCs. I think some of
>>> the needs there begin to have ui implications which could be
>>> high-friction. I really want to finish it someday, but I've been too
>>> busy lately at $job, and now my experiments are all rather stale.
>>>
>>> It would be a good discussion to have over at the git list (copied).
>>> Heiko and Jens have laid some new groundwork in this area and it may
>>> be a good time to revisit it.  Or maybe they've even moved deeper than
>>> that; I have been distracted for well over a year now.
>>>
>
> Glad you're working (or planning to) working on submodulues. This is
> also on my todo list for the next months as well.

More hands are always welcome!

> I'd review stuff in that area if you're looking for reviewers.

I'll be happy help too.

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

[Phil Hord]

On Tue, Jun 9, 2015 at 2:40 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 07.06.2015 um 08:26 schrieb Stefan Beller:
>>
>> On 06.06.2015 12:53, Luca Milanesio wrote:
>>>>
>>>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com>
>>>> wrote:
>>>>>
>>>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>>>> see a "git diff --recursive" that goes through the submodules as well
>>>>> :-)
>>>>>
>>>>> Something possibly to propose to the Git mailing list?
>
>
> Such an option makes lots of sense to me (though "--recurse-submodules"
> should be its name for consistency reasons). This could be an alias for
> "--submodule=full", as the "--submodule" option controls the format of
> submodule diffs.

To me, --recurse-submodules means submodules are still not first-class
citizens.  But let's put that aside for a moment; I don't care about
the switch name too much as long as I can configure
'diff.recurse-submodules = true'.

[The following is rather long.  I'm sorry for that.  Feel free to look
away when it gets too vague.]

Let me set up a submodule like so:

  $ git init /tmp/Super && cd /tmp/Super
  Super$ git submodule add https://github.com/gitster/git.git Foo

I wish to be able to grep from Super and find matches in all my submodules.

  Super$ git grep --recurse-submodules base--int
  Foo/.gitignore:/git-rebase--interactive
  Foo/Makefile:SCRIPT_LIB += git-rebase--interactive

But I want this to work naturally across git-module boundaries, so I
want this also to work (grepping a super-project from within a
submodule):

  Super$ cd Foo
  Foo$ git grep --recurse-submodules base--int ..
  .gitignore:/git-rebase--interactive
  Makefile:SCRIPT_LIB += git-rebase--interactive

I expect some groans from the audience here, because I think if the
syntax above worked, then so would this:

  $ cd /tmp
  tmp$ git grep base--int /tmp/Super/Foo
  /tmp/Super/Foo/.gitignore:/git-rebase--interactive
  /tmp/Super/Foo/Makefile:SCRIPT_LIB += git-rebase--interactive

This usage has nothing to do with submodules, really, except that it
allows git commands to reach into foreign git directories by virtue of
the path supplied as some argument instead of via $GITDIR, and in
doing so it helps solve some git submodules use cases of mine.

But if that did not turn your stomach, try this one:

  $ cd /tmp/Super
  Super$ printf "Some submodule data">Foo/data.txt
  Super$ git add Foo/data.txt
  fatal: Pathspec 'Foo/data.txt' is in submodule 'Foo'
  Super$ git add --recurse-submodules Foo/data.txt

Some notes on this usage:

1. --recurse-submodules seems like a reasonable name for this switch,
especially when you consider the 'git add --recurse-submodules .' use
case.

2. This recursive 'git add' seems dangerous to me unless git-status
also shows all the changed/untracked files in submodules as well if
the --recurse-submodules switch is included.  This would support the
expectation that 'git add .' is going to add the files shown by 'git
status .'

3. Configuring --recurse-submodules as the default mode for 'git add'
but not for 'git status' seems reckless enough that I think there
should not be separate options for these two commands.  There are
probably many other "cross-command" scenarios with similar coupling.

Moving on, as we have :/ to mean 'workdir root', I wonder how you
would spell "super-project workdir root".  Maybe it would be ::/

I realize the kinds of features I'm talking about require extensive
code changes in Git.  For example, consider the meaning of this:

  Super$ git diff --recurse-submodules origin/next origin/master

Since I created Super just a few minutes ago and it has no remote
named 'origin', this command seems meaningless to me.  But suppose
that origin/next and origin/master did exist in my Super project.
Then, I would expect in my wishlist Git, that

A.  Super$ git diff --recurse-submodules origin/next origin/master
This would include differences in Foo between origin/master:Foo and
origin/next:Foo; that is, the commits referenced from those gitlinks
in Super.

B.  Super$ git diff --recurse-submodules origin/next HEAD
This would include differences in Foo between origin/master:Foo and
HEAD:Foo; that is, the commits referenced from those gitlinks in
Super.

C.  Super$ git diff --recurse-submodules origin/next
This would include differences in Foo between origin/master:Foo and
the current Foo workdir.

D.  Super$ cd Foo && git diff origin/next
This would include differences in Foo between the Foo submodule's
origin/master and the current Foo workdir.

Now, C and D seem confusingly similar to me and technically very
different.  I could understand the results, but I could easily be led
astray, especially if I am writing a script.  But I still think it is
reasonable and correct.

I think this could have dire consequences for some commands like 'git
apply'. But I think it is reasonable for git apply to reject such
cross-project diffs, at least in the beginning.  :-)

While I am thinking about it, let me also mention these cases:
E.  Super$ git diff --recurse-submodules origin/next origin/master -- Foo
I think 'origin/next' and 'origin/master' here are referring to
Super's refs, but I can imagine an implementer choosing to use Foo's
instead.

F.  Super$ cd Foo
      Foo$ git diff --recurse-submodules origin/next origin/master -- ..
If this worked, I would think 'origin/next' and 'origin/master' here
must refer to Super's refs even though I began in Foo.  This one is so
ambiguous I think I would have to call this an error.  More
specifically, I think it would have to be rewritten like this next one
(G).

G.  Super$ cd Foo
      Foo$ git -C .. diff --recurse-submodules origin/next origin/master
That is, at least for 'git diff', the <path> parameter at the end is
only used to filter the results; it is not used to find the git-dir.

But look at me speaking in the present tense.  How silly.  I live too
much in my own imagination.

Phil

-- 

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

[Jens Lehmann]

Am 11.06.2015 um 18:11 schrieb Phil Hord:
> On Tue, Jun 9, 2015 at 2:40 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Am 07.06.2015 um 08:26 schrieb Stefan Beller:
>>>
>>> On 06.06.2015 12:53, Luca Milanesio wrote:
>>>>>
>>>>> On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
>>>>> On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Ideally, as a "git clone --recursive" already exists, I would like to
>>>>>> see a "git diff --recursive" that goes through the submodules as well
>>>>>> :-)
>>>>>>
>>>>>> Something possibly to propose to the Git mailing list?
>>
>>
>> Such an option makes lots of sense to me (though "--recurse-submodules"
>> should be its name for consistency reasons). This could be an alias for
>> "--submodule=full", as the "--submodule" option controls the format of
>> submodule diffs.
>
> To me, --recurse-submodules means submodules are still not first-class
> citizens.  But let's put that aside for a moment; I don't care about
> the switch name too much as long as I can configure
> 'diff.recurse-submodules = true'.

After somebody implemented the 'full' mode for 'diff --submodule',
setting 'diff.submodule' to 'full' would make --recurse-submodules the
default for diff (unless recursing into the submodules is overridden
by either the global 'diff.ignoreSubmodules' or the per-submodule
'submodule.<name>.ignore' setting of course).

> [The following is rather long.  I'm sorry for that.  Feel free to look
> away when it gets too vague.]

Sorry, that was too long for todays git time budget ;-)

Re: Submodules as first class citizens (was Re: Moving to subtrees for plugins?)

[Heiko Voigt]

On Tue, Jun 09, 2015 at 08:40:14PM +0200, Jens Lehmann wrote:
> Am 07.06.2015 um 08:26 schrieb Stefan Beller:
> >On 06.06.2015 12:53, Luca Milanesio wrote:
> >>>On 6 Jun 2015, at 18:49, Phil Hord <phil.hord@gmail.com> wrote:
> >>>On Fri, Jun 5, 2015, 2:58 AM lucamilanesio <luca.milanesio@gmail.com> wrote:
> >>>>Ideally, as a "git clone --recursive" already exists, I would like to
> >>>>see a "git diff --recursive" that goes through the submodules as well :-)
> >>>>
> >>>>Something possibly to propose to the Git mailing list?
> 
> Such an option makes lots of sense to me (though "--recurse-submodules"
> should be its name for consistency reasons). This could be an alias for
> "--submodule=full", as the "--submodule" option controls the format of
> submodule diffs.

BTW, for long running topics (or low hanging fruits) we collect/link
everything in the wiki of Jens git fork on github. This is the central
page:

https://github.com/jlehmann/git-submod-enhancements/wiki

Maybe everyone that has work in the queue can add his work there (the work that
takes more time) so we can avoid doubling any effort. Not everything there is
up to date at the moment but I will look into it to remove outdated things.

> >>>I've worked on git diff --recursive a bit myself, along with some
> >>>simpler use cases (git ls-tree --recursive) as POCs. I think some of
> >>>the needs there begin to have ui implications which could be
> >>>high-friction. I really want to finish it someday, but I've been too
> >>>busy lately at $job, and now my experiments are all rather stale.
> >>>
> >>>It would be a good discussion to have over at the git list (copied).
> >>>Heiko and Jens have laid some new groundwork in this area and it may
> >>>be a good time to revisit it.  Or maybe they've even moved deeper than
> >>>that; I have been distracted for well over a year now.
> >>>
> >
> >Glad you're working (or planning to) working on submodulues. This is
> >also on my todo list for the next months as well.
> 
> More hands are always welcome!
> 
> >I'd review stuff in that area if you're looking for reviewers.
> 
> I'll be happy help too.

Me too.

Cheers Heiko