how to update a submodule?

how to update a submodule?

[Oliver Kullmann]

Hello,

it would be very helpful if somebody could tell me the
supposed workflow how to update a submodule (I'm using version
1.7.3.2; the man-page doesn't say much about it).

I have on the same machine two versions of repository A,
say A1, A2, and both contain repository B as submodule, but are otherwise
unrelated.

In A1 we have B up-to-date, but not in A2 (and with up-to-date I mean
the master branch in the same state as in A1).

git submodule update

is of no help, since A1, A2 have no relation to each other, and "update"
assumes some relation; the man-page doesn't speak of options for update,
and since "git submodule update" runs through all submodules, it's hard
to see how that could work.

So I have to go into B in A2, and do some pull (I suppose), but that
doesn't work so easily; the man-page seems to speak of B in A2 is a "normal repository"
but that doesn't seem to be the case in the sense that by default 
we are on "no branch". So it seems one can't do anything there, since 
you can't merge "no branch" into some other branch. The branch "no branch"
is up-to-date after a pull, but this can't be communicated to the other
branches (so seems to be misleading, or a waste of time).

So apparently I have to checkout master in A2:B, and then pull from
master in A1:B ?

Would be good if somebody could shed light on this.

In general I find that "pull" is much under-documented in git.
What I would like to do in A2 is just to say "pull everything from A1",
which seems a logical thing to do, and which should include all corresponding
branches and all corresponding submodules, but such a "pull all" doesn't
seem to exist?

Thanks for your help in any case!

Oliver

P.S. What would be really needed for the Git documentation is to speak
about the fundamental ideas, about the "mental images".

The submodule documentation starts with a blurb-sentence which I would
regard as misleading.
Then comes a paragraph about what it is not.
Then comes some hints regarding the implementation.
Then the final paragraph seems to indicate that above I could just
use "git pull A2" for A1, and would get B in A2 from B in A1, but only not checked out?
This doesn't seem to be the case, since apparently then "git submodule update"
tries to fetch from some foreign repository (not just checking out what was fetched before)?

Adding here examples for a complete workflow (perhaps with various stages, but without
complications(!)), from initialisation to updates from other repositories would be
very helpful.

As you can see, I don't know what to expect, and so I don't know what are errors in
the usage, and what is supposed to be. No mental image about "submodule" is offered,
and I don't know what is its intention. A great thing would be using the computer-science
concept of an abstract data type, so to explain what the interface and its semantics is,
not what is the implementation.

Re: how to update a submodule?

[Seth Robertson]

In message <20101231222438.GA28199@cs-wsok.swansea.ac.uk>, Oliver Kullmann writ
es:

    it would be very helpful if somebody could tell me the
    supposed workflow how to update a submodule (I'm using version
    1.7.3.2; the man-page doesn't say much about it).

https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial

--------------------------------------------------
cd submodule_path
git checkout <branchname>
git pull
cd .. # or otherwise get to the superproject

git add submodule_path
git commit -m "Updated submodule to latest <branchname>"
git submodule update
--------------------------------------------------

This (untested) code obviously only works for a single repository.  If
you want to do it for all repositories, you need something more like
the untested:

--------------------------------------------------
git submodule foreach git checkout <branchname>
git submodule foreach git pull
git add -a -m "Updated all submodules to latest <branchname>"
git submodule update
--------------------------------------------------

If you find yourself doing this continuously, and doing it for all of
your submodules, you may way to use gitslave instead of submodules
which keeps the branches checked out all of the time so all you need to
do is `gits pull`.  This may be better, or worse, for you workflow.

gitslave (http://gitslave.sf.net) is useful when you control and
develop on the subprojects at more of less the same time as the
superproject, and furthermore when you typically want to tag, branch,
push, pull, etc all repositories at the same time.

git-submodule is better when you do not control the subprojects or
more specifically wish to fix the subproject at a specific revision
even as the subproject changes

					-Seth Robertson

Re: how to update a submodule?

[Oliver Kullmann]

On Fri, Dec 31, 2010 at 06:42:01PM -0500, Seth Robertson wrote:
> 
> In message <20101231222438.GA28199@cs-wsok.swansea.ac.uk>, Oliver Kullmann writ
> es:
> 
>     it would be very helpful if somebody could tell me the
>     supposed workflow how to update a submodule (I'm using version
>     1.7.3.2; the man-page doesn't say much about it).
> 
> https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial
>

thanks, that's definitely something.
 
> --------------------------------------------------
> cd submodule_path
> git checkout <branchname>
> git pull
> cd .. # or otherwise get to the superproject
> 
> git add submodule_path
> git commit -m "Updated submodule to latest <branchname>"
> git submodule update
> --------------------------------------------------
> 
> This (untested) code obviously only works for a single repository.  If
> you want to do it for all repositories, you need something more like
> the untested:
> 
> --------------------------------------------------
> git submodule foreach git checkout <branchname>
> git submodule foreach git pull
> git add -a -m "Updated all submodules to latest <branchname>"
> git submodule update
> --------------------------------------------------
> 

As far as I see that, this doesn't concern the problem how to I update
one repository with submodules from another repository with "these" submodules
(as the same paths)?

Actually, even the simplest case of just cloning a repository with submodules
doesn't work:
After cloning, "git submodule status" shows "-", okay so I do "git submodule init",
which already shows a false understanding --- it shows the URL of the old repository,
from which the original submodule originated, which is long gone and no longer relevant.
Then of course "git submodule update" fails, and in the submodule there is just nothing.

A clone should provide a relation to the mother-clone, not transitively to
the grandmother-clone.

The problem seems to be that the information about the place where to update a
submodule is in .gitmodules, which git actually has under version control (different
from other configuration information), and thus copies it verbatimly.
Okay, then apparently after a clone .gitmodules has to be updated (by hand).

So .gitmodules concerns only "git submodule update", not "git pull" from within
the submodules? This would be good to know, to understand the role of the information
in .gitmodules (where the task of "git submodule init" is apparently just to
transport this information to .git/configure ?).

This has the disadvantage, that one has to publish this private
information about the places where by chance one is pulling from?
Perhaps I should then put .gitmodules into .gitignore? Or would that have
bad consequences??

Or is the idea that .gitmodules normally is not under version control, and
thus "git init", which actually hides the information of .gitmodules ?

> If you find yourself doing this continuously, and doing it for all of
> your submodules, you may way to use gitslave instead of submodules
> which keeps the branches checked out all of the time so all you need to
> do is `git pull`.  This may be better, or worse, for you workflow.
> 
> gitslave (http://gitslave.sf.net) is useful when you control and
> develop on the subprojects at more of less the same time as the
> superproject, and furthermore when you typically want to tag, branch,
> push, pull, etc all repositories at the same time.
>

On the one hand, I always want to pull "everything" (inclusive the content of all submodules),
however the submodules are completely independent (they are from Github, or collaborations on
papers, and such things --- I just want to conveniently carry around all of that with me in
one go).
 
> git-submodule is better when you do not control the subprojects or
> more specifically wish to fix the subproject at a specific revision
> even as the subproject changes
>

This seems to be the case here, so it seemed to me that gitslave wouldn't be appropriate
here.

Thanks for your help!

Oliver
 

Re: how to update a submodule?

[Jens Lehmann]

Am 01.01.2011 21:39, schrieb Oliver Kullmann:
> On Fri, Dec 31, 2010 at 06:42:01PM -0500, Seth Robertson wrote:
> As far as I see that, this doesn't concern the problem how to I update
> one repository with submodules from another repository with "these" submodules
> (as the same paths)?

I'm not sure I completely understand your use case, but submodules are
repositories of their own, so they don't get updated by just pulling
a superproject into another containing the same submodule. The submodule
changes have to be pushed to its own parent repository and can then be
fetched from there into another superproject's submodule.

> Actually, even the simplest case of just cloning a repository with submodules
> doesn't work:
> After cloning, "git submodule status" shows "-", okay so I do "git submodule init",
> which already shows a false understanding --- it shows the URL of the old repository,
> from which the original submodule originated, which is long gone and no longer relevant.
> Then of course "git submodule update" fails, and in the submodule there is just nothing.

Then the URL of the submodule in the .gitmodules file is not up to
date. You can either fix it there or - if you only want to change it
locally - edit the .git/config after the "git submodule init" copied
it there.

> The problem seems to be that the information about the place where to update a
> submodule is in .gitmodules, which git actually has under version control (different
> from other configuration information), and thus copies it verbatimly.
> Okay, then apparently after a clone .gitmodules has to be updated (by hand).

No, the problem seems to be that the remote URL in the submodule has
been changed directly without updating it in the .gitmodules of the
superproject (the recommended way to do that is to change it in the
.gitmodules file and then use "git submodule sync" to activate it).

> So .gitmodules concerns only "git submodule update", not "git pull" from within
> the submodules? This would be good to know, to understand the role of the information
> in .gitmodules (where the task of "git submodule init" is apparently just to
> transport this information to .git/configure ?).
> 
> This has the disadvantage, that one has to publish this private
> information about the places where by chance one is pulling from?

Yes, the submodules URL must be available - at least for fetching -
to everyone who wants to clone your superproject (and all changes
made in the submodule which are committed in the superproject must
have been pushed there to be able to check them out in the clone).

> Perhaps I should then put .gitmodules into .gitignore? Or would that have
> bad consequences??

Yes, because then git won't know where the it can clone the submodules
from when you clone your superproject somewhere else.

Re: how to update a submodule?

[Oliver Kullmann]

On Sun, Jan 02, 2011 at 12:30:15PM +0100, Jens Lehmann wrote:
> Am 01.01.2011 21:39, schrieb Oliver Kullmann:
> > On Fri, Dec 31, 2010 at 06:42:01PM -0500, Seth Robertson wrote:
> > As far as I see that, this doesn't concern the problem how to I update
> > one repository with submodules from another repository with "these" submodules
> > (as the same paths)?
> 
> I'm not sure I completely understand your use case, but submodules are
> repositories of their own, so they don't get updated by just pulling
> a superproject into another containing the same submodule. The submodule
> changes have to be pushed to its own parent repository and can then be
> fetched from there into another superproject's submodule.
>

I know that --- but if there wouldn't be any savings possible (in terms of using it),
then submodules would be pointless, and so the question is *how* to use them.

I see that my use-case is not understood so I try it:

The good thing with Git is that there are no central repositories.
That's exactly what I want to use, but again and again the automatic
assumptions of "central repositories" are made, which should be actually alien to Git.

More concretely: For a collaboration on a scientific paper (where more and more
people have never seen each other, and this is also the case here) one doesn't
use version-control yet because of all this server-nonsense. Now with Git we can
just toss tarballs of git-archives around! Sent via e-mail (please note that zero
knowledge of git-remotes or whatever is required for that --- that's very important),
these archives float around, without any URL or central repository, but they
are locally merged and whatever. I think that's exactly exploiting the Git philosophy.

So I have a couple of those. Then I have modules to teach, where I organise also various
forms of collaborations with colleagues and students (various networks, various levels
of security).

Then there are software projects. Some private, some not.

All of that happens independently. I have at this time around 15 of such active projects
(some smaller, some rather big).

Now it happens that things are forgotten, updates aren't done, I just forget to copy
a repository from one place to another, etc.
Important that from my main working place at home to the main university working place
there is no network connection, but memory sticks do the job (which is fine and safe).

Sure, I can (and perhaps have to) start writing scripts, which just exploit basic git,
and add some form of super-organisation on top of it.

However I still hope that the task is so common that I can use tools.

To summarise: At some point in time on some machine, a super-repository A is updated,
from various (permanently changing) places etc. No problem with pull itself.
The submodules each needs to be taken care of individually.

Okay, but then I want to take that whole thing, the supermodule with the 15 (or more)
submodules, copy it to the memory stick, and on my workplace untar it (decrypt it ...),
and update a super-repository B *over there*, just from A, which has nothing to
do how I created A (on a different machine, completely different circumstances).

It seems to fully comply with the original Git philosophy, that I can make some
form of "super-pull" from B, getting everything from A --- without the assumption
of some central URL (repository), neither for the supermodule nor for the submodules.
Just pulling everything over, as it happens with ordinary "git pull".

 
> > Actually, even the simplest case of just cloning a repository with submodules
> > doesn't work:
> > After cloning, "git submodule status" shows "-", okay so I do "git submodule init",
> > which already shows a false understanding --- it shows the URL of the old repository,
> > from which the original submodule originated, which is long gone and no longer relevant.
> > Then of course "git submodule update" fails, and in the submodule there is just nothing.
> 
> Then the URL of the submodule in the .gitmodules file is not up to
> date. You can either fix it there or - if you only want to change it
> locally - edit the .git/config after the "git submodule init" copied
> it there.
>

That file .gitmodule seems to enforce the central structure, contradicting the original
(and great!) Git insight of fully distributed version control.
 
> > The problem seems to be that the information about the place where to update a
> > submodule is in .gitmodules, which git actually has under version control (different
> > from other configuration information), and thus copies it verbatimly.
> > Okay, then apparently after a clone .gitmodules has to be updated (by hand).
> 
> No, the problem seems to be that the remote URL in the submodule has
> been changed directly without updating it in the .gitmodules of the
> superproject (the recommended way to do that is to change it in the
> .gitmodules file and then use "git submodule sync" to activate it).
>

Again, I hope by now it is clear that in my cases it makes no sense
to attribute an URL to the submodule.
 
> > So .gitmodules concerns only "git submodule update", not "git pull" from within
> > the submodules? This would be good to know, to understand the role of the information
> > in .gitmodules (where the task of "git submodule init" is apparently just to
> > transport this information to .git/configure ?).
> > 
> > This has the disadvantage, that one has to publish this private
> > information about the places where by chance one is pulling from?
> 
> Yes, the submodules URL must be available - at least for fetching -
> to everyone who wants to clone your superproject (and all changes
> made in the submodule which are committed in the superproject must
> have been pushed there to be able to check them out in the clone).
>

Then submodules would be a wrong concept, as it seems to me, contradicting
decentralised version control. Still I have the hope that actually
submodule can be used, but apparently that file .gitmodules is in the way
(false information is much worse than no information!).
 
> > Perhaps I should then put .gitmodules into .gitignore? Or would that have
> > bad consequences??
> 
> Yes, because then git won't know where the it can clone the submodules
> from when you clone your superproject somewhere else.

But that would be what is needed --- there is no "somewhere else", but
just the given repository.

Puuuh, I really really tried hard now to make my use-case clear :-|. Hopefully
now the picture emerges.

Thanks for your considerations! And I hope you don't mind me speaking of
"contradictions" etc., but I feel there are fundamental misunderstandings.

Thanks!

Oliver

Re: how to update a submodule?

[Jens Lehmann]

Am 02.01.2011 16:55, schrieb Oliver Kullmann:
> On Sun, Jan 02, 2011 at 12:30:15PM +0100, Jens Lehmann wrote:
>> Am 01.01.2011 21:39, schrieb Oliver Kullmann:
>>> On Fri, Dec 31, 2010 at 06:42:01PM -0500, Seth Robertson wrote:
>>> As far as I see that, this doesn't concern the problem how to I update
>>> one repository with submodules from another repository with "these" submodules
>>> (as the same paths)?
>>
>> I'm not sure I completely understand your use case, but submodules are
>> repositories of their own, so they don't get updated by just pulling
>> a superproject into another containing the same submodule. The submodule
>> changes have to be pushed to its own parent repository and can then be
>> fetched from there into another superproject's submodule.
> 
> I know that --- but if there wouldn't be any savings possible (in terms of using it),
> then submodules would be pointless, and so the question is *how* to use them.

No, they aren't pointless at all. But if you want to collaborate using
submodules they IMO work best if all your coworkers are able to access
the same submodule repos you are pushing to. Otherwise you'll have to
transport all submodule changes additionally to those of the superproject
(which might be more of a hassle than not using submodules in the first
place). Then you might be better off pulling the modules into your repo
using "git subtree" or "gitslave".

A possibility to put all submodule commits in the object directory of
the superproject has been discussed some time ago on this list [1] and
at the last GitTogether. That might be just what you need, but I am not
aware of any work done in that direction yet.

> The good thing with Git is that there are no central repositories.
> That's exactly what I want to use, but again and again the automatic
> assumptions of "central repositories" are made, which should be actually alien to Git.

No, Git works perfectly fine with central repositories too (and that is
a feature :-). But I think I understand where your impression comes from.
Submodules don't work very well when you change URLs (that can result
in forcing your coworkers to do a "git submodule sync" in their repo
every time they switch to a commit with a changed URL). But while that
somehow works not being able to access a submodules repo doesn't work at
all. So the constraint for submodules is to have a repo which is visible
for the people you work with.

But submodules don't really force you into the centralized model, as you
can modify the .gitmodules file e.g. in a downstream fork and let it point
to your own forked version of the submodules repos where you can do your
own development independent of the submodules upstream.

> Puuuh, I really really tried hard now to make my use-case clear :-|. Hopefully
> now the picture emerges.

Yep, thanks for sharing that!


[1] http://thread.gmane.org/gmane.comp.version-control.git/151473/

Re: how to update a submodule?

[Jonathan Nieder]

Jens Lehmann wrote:

> Submodules don't work very well when you change URLs (that can result
> in forcing your coworkers to do a "git submodule sync" in their repo
> every time they switch to a commit with a changed URL).

For the future, it is probably most convenient to use URLs starting
with "../" or "./", which would rarely need to change.  That can even
work with passing around a tarball of bundles and a script to
reconstitute them, I think.

Thanks for some food for thought.
Jonathan