Why do git submodules require manual checkouts and commits?

Why do git submodules require manual checkouts and commits?

[skillzero]

Why do git submodules require manually committing the submodule itself
to each super repository after something in the submodule repository
changes? Is there some reason the super repository can't just "link"
to the submodules by branch name? It seems that if the .gitsubmodules
also specified the branch to use:

[submodule "libfoo"]
	path = libs/foo
	url = git://foo.com/git/libfoo.git
	branch = master

[submodule "libbar"]
	path = libs/bar
	url = git://bar.com/git/libbar.git
	branch = stable

Then a git pull (or git clone) of the super repository could also pull
in all submodules. A commit to a file in a submodule would then be
automatically reflected in the super repository (since the super
repository would always pull HEAD of that branch).

Is this difficult (or somehow undesirable)?

Re: Why do git submodules require manual checkouts and commits?

[Johannes Schindelin]

Hi,

On Thu, 15 May 2008, skillzero@gmail.com wrote:

> Why do git submodules require manually committing the submodule itself
> to each super repository after something in the submodule repository
> changes?

Submodules are special.

You cannot recreate the exact state from a commit in the superproject, for 
one, and often not even from the commit itself, since the submodule can 
contain more than just the tracked files.

Also, no submodule _has_ to be checked out.  If you are working inside a 
superproject, chances are that you are uninterested in most of the 
submodules.

So no, there is nothing to change here, please move along.

Ciao,
Dscho

Re: Why do git submodules require manual checkouts and commits?

[Avery Pennarun]

On 5/16/08, skillzero@gmail.com <skillzero@gmail.com> wrote:
> Why do git submodules require manually committing the submodule itself
>  to each super repository after something in the submodule repository
>  changes? Is there some reason the super repository can't just "link"
>  to the submodules by branch name? It seems that if the .gitsubmodules
>  also specified the branch to use:

This is difficult but (contrary to what others might say :)) in my
opinion it's worth fixing.  Exactly how to fix it is an open question
though.

The main reason the simple approach you suggested (just link to a
branch instead of a particular commit) isn't good is that it doesn't
guarantee you always get the same version every time.  If the
supermodule links to submodule branch "foo" and makes supermodule
commit 23abc918c, and someone later pushes to submodule branch "foo",
then checking out commit 23abc918c in the supermodule would get the
very latest submodule "foo", not the one you had when 23abc918c was
created.  Thus all sorts of bad things could happen.

So I think it would be very bad if the supermodule automatically
updated to the latest version of the submodule whenever you commit in
the submodule.  *However*, the other way around might be fine: if you
commit in the supermodule, maybe it should commit in the submodule at
the same time and link to that specific commit.  I'm pretty sure that
idea doesn't have any *fundamental* flaws, it's just got a lot of
really tricky details that need to be worked out.  I've seen some
"recursive commit" and "recursive push" patches floating around, so
people are actively working on this.  One of the hardest things to
deal with is where to auto-push submodules to (which remote? which
branch?) when you push the supermodule.

Have fun,

Avery

Re: Why do git submodules require manual checkouts and commits?

[Johannes Schindelin]

Hi,

On Fri, 16 May 2008, Avery Pennarun wrote:

> So I think it would be very bad if the supermodule automatically
> updated to the latest version of the submodule whenever you commit in
> the submodule.  *However*, the other way around might be fine: if you
> commit in the supermodule, maybe it should commit in the submodule at
> the same time and link to that specific commit.  I'm pretty sure that
> idea doesn't have any *fundamental* flaws, it's just got a lot of
> really tricky details that need to be worked out.

Just the fundamental flaw that you might _not_ want to commit that, just 
as you can have a dirty Makefile _forever_.

Ciao,
Dscho

Re: Why do git submodules require manual checkouts and commits?

[Avery Pennarun]

On 5/16/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  On Fri, 16 May 2008, Avery Pennarun wrote:
>  > So I think it would be very bad if the supermodule automatically
>  > updated to the latest version of the submodule whenever you commit in
>  > the submodule.  *However*, the other way around might be fine: if you
>  > commit in the supermodule, maybe it should commit in the submodule at
>  > the same time and link to that specific commit.  I'm pretty sure that
>  > idea doesn't have any *fundamental* flaws, it's just got a lot of
>  > really tricky details that need to be worked out.
>
> Just the fundamental flaw that you might _not_ want to commit that, just
>  as you can have a dirty Makefile _forever_.

I consider that one of the annoying details rather than a fundamental
flaw.  I agree that it's hard to solve though.

Think of it this way: I can commit, or not commit, my dirty Makefile
at the same time as everything else (in a single project) with a
single "git commit" line, depending on what I want to do.  Things like
"git commit -a" and "git add -u" speed up the common case where I just
want to commit everything.  But with submodules, that common case
looks more like this:

   cd sub
   git checkout -b manual_branchname_because_there_was_no_default
   git commit -a
   git push etc.
   cd ..
   git commit -a
   git push etc.

That's *really* tedious, and the number of commands multiplies when
you have more than one submodule going at once.

I think git's submodules are awesome because they *don't* have
fundamental flaws.  They just need an (optional) more automated
workflow for the common case.  And I'll be sure to propose one when I
figure out what my common case actually is :)

Have fun,

Avery

Re: Why do git submodules require manual checkouts and commits?

[Johannes Schindelin]

Hi,

On Fri, 16 May 2008, Avery Pennarun wrote:

> Think of it this way: I can commit, or not commit, my dirty Makefile at 
> the same time as everything else (in a single project) with a single 
> "git commit" line, depending on what I want to do.  Things like "git 
> commit -a" and "git add -u" speed up the common case where I just want 
> to commit everything.  But with submodules, that common case looks more 
> like this:
> 
>    cd sub
>    git checkout -b manual_branchname_because_there_was_no_default
>    git commit -a
>    git push etc.
>    cd ..
>    git commit -a
>    git push etc.

Funny, for me it looks completely different:

$ cd sub
# work, work, work
# from time to time commit
# from time to time rebase -i to clean up some things
# test, test, test
# sometimes push

And then, every once in a while, it is

$ cd ..
$ git add submodule
$ git commit -s submodule
$ git push

> That's *really* tedious, and the number of commands multiplies when you 
> have more than one submodule going at once.

But hey, if you find that tedious, why did I not see a patch from you yet, 
implementing "git submodule commit-n-push"?

Ciao,
Dscho

Re: Why do git submodules require manual checkouts and commits?

[Avery Pennarun]

On 5/16/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  On Fri, 16 May 2008, Avery Pennarun wrote:
>  > But with submodules, that common case looks more
>  > like this:
>  >    cd sub
>  >    git checkout -b manual_branchname_because_there_was_no_default
>  >    git commit -a
>  >    git push etc.
>  >    cd ..
>  >    git commit -a
>  >    git push etc.
>
> Funny, for me it looks completely different:
>
>  $ cd sub
>  # work, work, work
>  # from time to time commit
>  # from time to time rebase -i to clean up some things
>  # test, test, test
>  # sometimes push
>
>  And then, every once in a while, it is
>
>  $ cd ..
>  $ git add submodule
>  $ git commit -s submodule
>  $ git push

Indeed.  I think you and I use submodules for very different purposes.
 The important case for me is that I often add a function to a library
(submodule), then *immediately* move on to use that function in an
application (supermodule).  When I commit the new submodule gitlink,
there are almost always other source code changes in the supermodule
at the same time.  And then I want to share the new application with
my co-workers shortly thereafter, often several times per day.

It looks like your own work on submodules tends to be quite isolated
from the supermodule, which would make sense if you use submodules for
(say) bundling a bunch of separate applications together.

>  > That's *really* tedious, and the number of commands multiplies when you
>  > have more than one submodule going at once.
>
> But hey, if you find that tedious, why did I not see a patch from you yet,
>  implementing "git submodule commit-n-push"?

Well, I think someone already submitted some "git submodule recursive" stuff.

I don't mean to complain.  I think the existing git-submodule code
gives all the raw materials necessary to solve all my
submodule-related problems, whatever those problems may be.  I'd much
rather have tedious-but-flexible instead of quick-but-inflexible.
However, having such powerful tools means it takes longer to figure
out how to use them most effectively.

Don't worry, I'll be the first to submit an all-singing all-dancing
shell script as soon as I figure out what that looks like.  Err,
unless someone else submits it first.  I guess that should cover all
my bases :)

(And now... I'm going to disappear for at least a week for vacation.  Bye!)

Have fun,

Avery

Re: Why do git submodules require manual checkouts and commits?

[Lars Hjemli]

On Fri, May 16, 2008 at 4:24 PM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>  On Fri, 16 May 2008, Avery Pennarun wrote:
>
>
> > Think of it this way: I can commit, or not commit, my dirty Makefile at
>  > the same time as everything else (in a single project) with a single
>  > "git commit" line, depending on what I want to do.  Things like "git
>  > commit -a" and "git add -u" speed up the common case where I just want
>  > to commit everything.  But with submodules, that common case looks more
>  > like this:
>  >
>  >    cd sub
>  >    git checkout -b manual_branchname_because_there_was_no_default
>  >    git commit -a
>  >    git push etc.
>  >    cd ..
>  >    git commit -a
>  >    git push etc.
>
>  Funny, for me it looks completely different:
>
>  $ cd sub
>  # work, work, work
>  # from time to time commit
>  # from time to time rebase -i to clean up some things
>  # test, test, test
>  # sometimes push
>
>  And then, every once in a while, it is
>
>  $ cd ..
>  $ git add submodule
>  $ git commit -s submodule
>  $ git push

Just to add to the picture, for me it's

$ (cd submodule && git checkout tag)
$ git add submodule
$ git commit -s -m "Use submodule-tag"

If I want to work in/with the submodule, I usually do that by "cd
../submodule", i.e. I've got another clone of the submodule
repository.

--
lh

Re: Why do git submodules require manual checkouts and commits?

[skillzero]

On Fri, May 16, 2008 at 6:43 AM, Avery Pennarun <apenwarr@gmail.com> wrote:

> The main reason the simple approach you suggested (just link to a
> branch instead of a particular commit) isn't good is that it doesn't
> guarantee you always get the same version every time.  If the
> supermodule links to submodule branch "foo" and makes supermodule
> commit 23abc918c, and someone later pushes to submodule branch "foo",
> then checking out commit 23abc918c in the supermodule would get the
> very latest submodule "foo", not the one you had when 23abc918c was
> created.  Thus all sorts of bad things could happen.

It seems like this could be handled by changing the way a commit is
tracked such that it also includes the commit hash of the submodules.
So for example, instead of refs/heads/master just being just a single
commit hash, it would be a list:

[refs/heads/master]
       1033bb1ed64d1dbac9f93360e69402195386d145
       libfoo = a13dcb7f26a160d85038385b3024b700dec208d9
       libbar = 123123123dcb7f26a160d85038385b3024b700de

If I commit a change to libfoo, it updates libfoo's commit hash to the latest.
If I do git pull on the supermodule, it recursively does git pull on
all submodules and then updates refs/heads/master (or whatever branch
you happen to be on) with the new commit hashes of the submodules.

If you checkout supermodule commit 1033bb1, it would look at the
supermodule's refs/heads/master for 1033bb and see that libfoo was at
a13dcb and libbar was at 123123 and it would then check out those
commits, respectively.

If you branch the supermodule, it recursively branches the submodules
with the same name, but using a namespace. Maybe super/<branch>
similar to how origin/branch is use for remote branches.

If you push the supermodule, it would also recursively push the
submodules using the URL specified when it was cloned (i.e. the one
specified in the .gitsubmodules file).

What I'm trying to do is use git to manage multiple, large
repositories that share a lot of code. I want to make things more
sparse so independent pieces can be used standalone or included as
submodules in other, larger projects.