Submodules

Submodules

[Dmitry Kakurin]

Where can I find more information about submodules?
I'm interested in both internal implementation details and how to use them.

- Dmitry

Re: Submodules

[Johannes Schindelin]

Hi,

On Tue, 7 Aug 2007, Dmitry Kakurin wrote:

> Where can I find more information about submodules?
> I'm interested in both internal implementation details and how to use them.

Most information is found in git-submodule.sh, although 
Documentation/git-submodule.txt has beginnings of a good documentation.

The basic idea is this:

A subdirectory contains a self-contained project.

Once that project is at a stage you want to have in the superproject, you 
can commit that submodule state, much like a subdirectory.

However, the history can progress in much coarser steps for the 
superproject.  IOW an ideal case for msysgit.git, where we want to have a 
stable version which can be checked out with GitMe.

Also, find more about submodules here:

http://git.or.cz/gitwiki/SubprojectSupport?highlight=%28submodule%29

Ciao,
Dscho

Re: Submodules

[Johannes Schindelin]

Hi,

completely forgot: submodules do not need to be checked out.  So, if there 
is a submodule you are not interested in working on, you can leave it.

Ciao,
Dscho

Re: Submodules

[Dmitry Kakurin]

Yeah, I've already went thru these. But I want (much) more details.
I can ask a bunch of random questions on this list, like:

* Why after 'submodule update' it becomes detached, what if I want it
to stay on certain branch?
* How do I control which branches are fetched?
* What if I do 'commit -a' standing in /super, will /super/submodule
commit as well?
* What if I'm standing in /super/submodule and do 'commit -a'? Will
super notice it in any way?

But I was hoping that it's all summarized somewhere already.

- Dmitry
On 8/7/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Tue, 7 Aug 2007, Dmitry Kakurin wrote:
>
> > Where can I find more information about submodules?
> > I'm interested in both internal implementation details and how to use them.
>
> Most information is found in git-submodule.sh, although
> Documentation/git-submodule.txt has beginnings of a good documentation.
>
> The basic idea is this:
>
> A subdirectory contains a self-contained project.
>
> Once that project is at a stage you want to have in the superproject, you
> can commit that submodule state, much like a subdirectory.
>
> However, the history can progress in much coarser steps for the
> superproject.  IOW an ideal case for msysgit.git, where we want to have a
> stable version which can be checked out with GitMe.
>
> Also, find more about submodules here:
>
> http://git.or.cz/gitwiki/SubprojectSupport?highlight=%28submodule%29
>
> Ciao,
> Dscho

Re: Submodules

[Dmitry Kakurin]

This is exactly the level of details I'm talking about:
* how come sumbodules are not initialized when I do a clone of super.
I expect to be able to build super after I clone it. Is there a new
(undocumented) flag to clone?
* is it OK to *not* init a submodule? will super become unhappy? Can I
do commits to super in this case?
* why submodules should be listed in 2 places: in .submodules and in
super/.git/config?

- Dmitry

P.S. There is an old saying "One fool can ask so many questions that a
thousand wise man would not be able to answer" :-)

On 8/7/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> completely forgot: submodules do not need to be checked out.  So, if there
> is a submodule you are not interested in working on, you can leave it.

Re: Submodules

[Johannes Schindelin]

Hi,

On Tue, 7 Aug 2007, Dmitry Kakurin wrote:

> * how come sumbodules are not initialized when I do a clone of super.
> I expect to be able to build super after I clone it. Is there a new
> (undocumented) flag to clone?

There might be.  But one of the reasons submodules are not just tree 
objects is that you should be able _not_ to check out all of them (think 
KDE).

> * why submodules should be listed in 2 places: in .submodules and in
> super/.git/config?

Well, URLs can change.  Therefore, .submodules has what is considered the 
current URL, but you can change it yourself in .git/config.  (At least 
that is how I understood it.)

Ciao,
Dscho

Re: Submodules

[Sven Verdoolaege]

On Tue, Aug 07, 2007 at 04:10:38PM -0700, Dmitry Kakurin wrote:
> Yeah, I've already went thru these. But I want (much) more details.
> I can ask a bunch of random questions on this list, like:
> 
> * Why after 'submodule update' it becomes detached, what if I want it
> to stay on certain branch?

You'd need to have a way to specify which branch you want to have updated.
There was some discussion about this some time ago and the general
conclusion was that a detached HEAD was the best solution.

> * How do I control which branches are fetched?

Currently, you can't.

> * What if I do 'commit -a' standing in /super, will /super/submodule
> commit as well?

No.

> * What if I'm standing in /super/submodule and do 'commit -a'? Will
> super notice it in any way?

Any commit in the submodule will be seen as a "change" to the submodule
from the perspective of the supermodule.

skimo

Re: Submodules

[Sven Verdoolaege]

On Tue, Aug 07, 2007 at 04:31:57PM -0700, Dmitry Kakurin wrote:
> This is exactly the level of details I'm talking about:
> * how come sumbodules are not initialized when I do a clone of super.

See you second question.

> I expect to be able to build super after I clone it. Is there a new
> (undocumented) flag to clone?

Not (yet).  Right now, you have to do

git submodule init
git submodule update

after you clone to fetch and check-out all (first-level) submodules.

> * is it OK to *not* init a submodule? will super become unhappy? Can I
> do commits to super in this case?

Yes. No. Yes.
In fact, only the "git submodule" subcommand are affected by a
"git submodule init".  Doing a "git submodule update" will actually
check out the submodules and from then on, the HEAD of this checked-out
submodule will be considered the content of the submodule in
the working tree of the supermodule.

> * why submodules should be listed in 2 places: in .submodules and in
> super/.git/config?

It only has to be specified in .git/config.
The value in .gitmodules (if present) is used by "git submodule init"
as a default value for the one in .git/config.

skimo

Re: Submodules

[Junio C Hamano]

There was a discussion about the semantics currently
implemented, as a subset of what we might want to eventually
have, and some recommended that we mention where things stand
now in the release notes.

How about this?

diff --git a/Documentation/RelNotes-1.5.3.txt b/Documentation/RelNotes-1.5.3.txt
index 21bb1fc..901f813 100644
--- a/Documentation/RelNotes-1.5.3.txt
+++ b/Documentation/RelNotes-1.5.3.txt
@@ -9,6 +9,19 @@ Updates since v1.5.2
 
 * The submodule support has Porcelain layer.
 
+  Note that the current submodule support is minimal and this is
+  deliberately so.  A design decision we made is that operations
+  at the supermodule level do not recurse into submodules by
+  default.  The expectation is that later we would add a
+  mechanism to tell git which submodules the user is interested
+  in, and this information might be used to determine the
+  recursive behaviour of certain commands (e.g. "git checkout"
+  and "git diff"), but currently we haven't agreed on what that
+  mechanism should look like.  Therefore, if you use submodules,
+  you would probably need "git submodule update" on the
+  submodules you care about after running a "git checkout" at
+  the supermodule level.
+
 * There are a handful pack-objects changes to help you cope better
   with repositories with pathologically large blobs in them.
 

Re: Submodules

[Johannes Schindelin]

Hi,

On Wed, 8 Aug 2007, Junio C Hamano wrote:

> +  Note that the current submodule support is minimal and this is
> +  deliberately so.  A design decision we made is that operations
> +  at the supermodule level do not recurse into submodules by
> +  default.  The expectation is that later we would add a
> +  mechanism to tell git which submodules the user is interested
> +  in, and this information might be used to determine the
> +  recursive behaviour of certain commands (e.g. "git checkout"
> +  and "git diff"), but currently we haven't agreed on what that
> +  mechanism should look like.  Therefore, if you use submodules,
> +  you would probably need "git submodule update" on the
> +  submodules you care about after running a "git checkout" at
> +  the supermodule level.
> +

Nice write up!

Is it true that you can run "update" without "init" first?  (I haven't 
tried yet.)

Also, I realised that git is not really happy unless you install 
alternates pointing to the object stores of the submodules.  Shouldn't we 
make this my default (for example in "init")?

Ciao,
Dscho

Re: Submodules

[Sven Verdoolaege]

On Wed, Aug 08, 2007 at 12:33:26PM -0700, Junio C Hamano wrote:
> +  Note that the current submodule support is minimal and this is
> +  deliberately so.  A design decision we made is that operations
> +  at the supermodule level do not recurse into submodules by
> +  default.  The expectation is that later we would add a
> +  mechanism to tell git which submodules the user is interested
> +  in, and this information might be used to determine the
> +  recursive behaviour of certain commands (e.g. "git checkout"
> +  and "git diff"), but currently we haven't agreed on what that
> +  mechanism should look like.  Therefore, if you use submodules,
> +  you would probably need "git submodule update" on the
> +  submodules you care about after running a "git checkout" at
> +  the supermodule level.
> +

Maybe you could be a bit more explicit about the
possibly unexpected results of "git checkout".
Perhaps something like the following instead of the
last sentence:

    In particular, if you have any submodules checked out,
    running a "git checkout" at the supermodule level will
    not update these submodules.  They will therefore appear
    to be modified (to the state prior to the checkout)
    to any subsequent git command, until they have been
    updated explicitly using "git submodule update".

skimo

Re: Submodules

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi,
>
> On Wed, 8 Aug 2007, Junio C Hamano wrote:
>
>> +  Note that the current submodule support is minimal and this is
>> +  deliberately so.  A design decision we made is that operations
>> +  at the supermodule level do not recurse into submodules by
>> +  default.  The expectation is that later we would add a
>> +  mechanism to tell git which submodules the user is interested
>> +  in, and this information might be used to determine the
>> +  recursive behaviour of certain commands (e.g. "git checkout"
>> +  and "git diff"), but currently we haven't agreed on what that
>> +  mechanism should look like.  Therefore, if you use submodules,
>> +  you would probably need "git submodule update" on the
>> +  submodules you care about after running a "git checkout" at
>> +  the supermodule level.
>> +
>
> Nice write up!
>
> Is it true that you can run "update" without "init" first?  (I haven't 
> tried yet.)
>
> Also, I realised that git is not really happy unless you install 
> alternates pointing to the object stores of the submodules.  Shouldn't we 
> make this my default (for example in "init")?

Sorry, but I do not use submodules yet myself.  The questions
need to be answered, errors need to be corrected and omissions
need to be added by people who have been more involved in that
area of the system.

That's would be Sven and Lars, probably.

I also think we should have a paragraph of similar spirit
about the work-tree stuff as the top-level bullet point, before
the "New commands and options" list.  I suspect Matthias and you
are the prime candidates to do the honors?

Re: Submodules

[Sven Verdoolaege]

On Wed, Aug 08, 2007 at 12:59:30PM -0700, Junio C Hamano wrote:
> Sorry, but I do not use submodules yet myself.

If you would ever start using submodules (for, say, gitgui),
then I'm sure we'd have much more usable submodules support
in no time.

skimo

Re: Submodules

[Junio C Hamano]

Ok, this is my second try.

 - Incorporated Sven's suggestion to give an example of
   problematic situation on submodule after "git-checkout" at
   the supermodule level;

 - Describe semantic clean-up of specifying GIT_DIR that is
   not strictly needed (because it is situated at the normal
   place relative to the working tree).

---
diff --git a/Documentation/RelNotes-1.5.3.txt b/Documentation/RelNotes-1.5.3.txt
index 21bb1fc..bf7a341 100644
--- a/Documentation/RelNotes-1.5.3.txt
+++ b/Documentation/RelNotes-1.5.3.txt
@@ -9,6 +9,40 @@ Updates since v1.5.2
 
 * The submodule support has Porcelain layer.
 
+  Note that the current submodule support is minimal and this is
+  deliberately so.  A design decision we made is that operations
+  at the supermodule level do not recurse into submodules by
+  default.  The expectation is that later we would add a
+  mechanism to tell git which submodules the user is interested
+  in, and this information might be used to determine the
+  recursive behaviour of certain commands (e.g. "git checkout"
+  and "git diff"), but currently we haven't agreed on what that
+  mechanism should look like.  In particular, if you have any
+  submodules checked out, running a "git checkout" at the
+  supermodule level will not update these submodules.  They will
+  therefore appear to be modified (to the state prior to the
+  checkout) to any subsequent git command, until they have been
+  updated explicitly using "git submodule update".
+
+* You can have a work tree not at the parent directory of .git
+  which is the repository.  This is done with GIT_WORK_TREE
+  environment, or --work-tree option.
+
+  This support slightly changes the semantics of having GIT_DIR
+  environment variable.  The rule used to be that when it is
+  given, you are supposed to be at the top level of the working
+  tree.  However, if it names a .git directory at the top of the
+  working tree, you can be in a subdirectory of that directory
+  and git notices you are in the subdirectory.  E.g. in t/
+  subdirectory of git.git project, 
+
+	$ cd t
+	$ GIT_DIR=../.git/ git ls-files -s
+
+  will list the files in t/ (i.e. your current directory).  It
+  used not to notice that you are in a subdirectory and listed
+  the files from the top-level of the working tree.
+
 * There are a handful pack-objects changes to help you cope better
   with repositories with pathologically large blobs in them.
 

Re: Submodules

[Andy Parkins]

On Wednesday 2007, August 08, Johannes Schindelin wrote:

> Also, I realised that git is not really happy unless you install
> alternates pointing to the object stores of the submodules.  Shouldn't we
> make this my default (for example in "init")?

I use submodules on a daily basis and haven't experienced this at all.  
Would you mind saying a little more about this problem?


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: Submodules

[Johannes Schindelin]

Hi,

On Wed, 8 Aug 2007, Andy Parkins wrote:

> On Wednesday 2007, August 08, Johannes Schindelin wrote:
> 
> > Also, I realised that git is not really happy unless you install
> > alternates pointing to the object stores of the submodules.  Shouldn't we
> > make this my default (for example in "init")?
> 
> I use submodules on a daily basis and haven't experienced this at all.  
> Would you mind saying a little more about this problem?

Not at all!

This is what I did.  I cloned msysgit.git, which contains one submodule.  
To initialise that, I did "git submodule init" and "git submodule update".  
It went and cloned the submodule.  Fine.

Then I committed in the submodule, a well-needed fix.

If I now go to the superproject again, and say "git submodule status", it 
will not find the newest commit, and complain that it cannot access that 
object.

Ciao,
Dscho

Re: Submodules

[Junio C Hamano]

Sven Verdoolaege <skimo@kotnet.org> writes:

> On Wed, Aug 08, 2007 at 12:59:30PM -0700, Junio C Hamano wrote:
>> Sorry, but I do not use submodules yet myself.
>
> If you would ever start using submodules (for, say, gitgui),
> then I'm sure we'd have much more usable submodules support
> in no time.

Heh, I do not have infinite time to spend on every little corner
features, and have been hoping that other people who have more
immediate need of submodule support to be the guinea pigs to
encounter real-life issues first and come up with solutions to
them, before I start using it.

So far, I think the plan has been working out alright ;-).

Re: Submodules

[Johannes Schindelin]

Hi,

On Wed, 8 Aug 2007, Junio C Hamano wrote:

> Sven Verdoolaege <skimo@kotnet.org> writes:
> 
> > On Wed, Aug 08, 2007 at 12:59:30PM -0700, Junio C Hamano wrote:
> >> Sorry, but I do not use submodules yet myself.
> >
> > If you would ever start using submodules (for, say, gitgui),
> > then I'm sure we'd have much more usable submodules support
> > in no time.
> 
> Heh, I do not have infinite time to spend on every little corner
> features, and have been hoping that other people who have more
> immediate need of submodule support to be the guinea pigs to
> encounter real-life issues first and come up with solutions to
> them, before I start using it.
> 
> So far, I think the plan has been working out alright ;-).

Sure.  I already like submodules, and I think that msysgit.git using it 
can bring submodules along more painlessly for git.git users.

Ciao,
Dscho

Re: Submodules

[Sven Verdoolaege]

On Wed, Aug 08, 2007 at 10:12:52PM +0100, Johannes Schindelin wrote:
> This is what I did.  I cloned msysgit.git, which contains one submodule.  
> To initialise that, I did "git submodule init" and "git submodule update".  
> It went and cloned the submodule.  Fine.
> 
> Then I committed in the submodule, a well-needed fix.
> 
> If I now go to the superproject again, and say "git submodule status", it 
> will not find the newest commit, and complain that it cannot access that 
> object.

There must be something else you're not telling us.
This procedure works just fine for me

bash-3.00$ git clone git://repo.or.cz/msysgit.git
Initialized empty Git repository in /home/skimo/git/msysgit/.git/
remote: Generating pack...
remote: Done counting 1857 objects.
remote: Deltifying 1857 objects...
remote:  100% (1857/1857) done
Indexing 1857 objects...
remote: Total 1857 (delta 475), reused 1857 (delta 475)
 100% (1857/1857) done
Resolving 475 deltas...
 100% (475/475) done
bash-3.00$ cd msysgit/
bash-3.00$ git submodule init
Submodule 'git' (git://repo.or.cz/git/mingw/4msysgit.git/) registered for path 'git'
bash-3.00$ git submodule update
Initialized empty Git repository in /home/skimo/git/msysgit/git/.git/
remote: Generating pack...
remote: Done counting 58074 objects.
remote: Deltifying 58074 objects...
remote:  100% (58074/58074) done
Indexing 58074 objects...
remote: Total 58074 (delta 40790), reused 55132 (delta 37853)
 100% (58074/58074) done
Resolving 40790 deltas...
 100% (40790/40790) done
Submodule path 'git': checked out 'f1e1dc5119bd9862bb4d2a975c8ca6362ea43af5'
bash-3.00$ cd git/
bash-3.00$ touch a; git add a; git commit -m 'add a'
Created commit 5318784: add a
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 a
bash-3.00$ cd ..
bash-3.00$ git submodule status
+5318784e6d492d988284a4d9aa4025496f2a3309 git (v1.5.3-rc2-690-g5318784)

skimo

submodules

[shawn wilson]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

So, I actually have another question I wasn't able to get to in this
example (which has color - sorry - less -F displays it decently)

What is shown here is that trying to add submodules in this repo
doesn't add the .gitmodules file - I can do it manually, but why isn't
it adding the file? There's a .git/modules directory populated with
the right stuff, but no .gitmodules file.

The initial question I was trying to demonstrate was that I've got a
repo with submodules. When I push branches to most of the modules, a:
git branch -r shows them for everyone. However, in one repo/module (I
think it's a repo created with git --bare --shared) no one else can
see (or pull) the remote branches and if I make a new clone of that
repo as myself, I can't see them either. However, those branches are
there and if I check that repo out on its own (not as a submodule of
the main repo) I and everyone else can see those remote branches.

This is git 1.8.2.1 btw.

[-- Attachment #2: typescript --]
[-- Type: application/octet-stream, Size: 51608 bytes --]

Re: submodules

[Jens Lehmann]

Am 03.05.2013 15:45, schrieb shawn wilson:
> So, I actually have another question I wasn't able to get to in this
> example (which has color - sorry - less -F displays it decently)
> 
> What is shown here is that trying to add submodules in this repo
> doesn't add the .gitmodules file - I can do it manually, but why isn't
> it adding the file? There's a .git/modules directory populated with
> the right stuff, but no .gitmodules file.
> 
> The initial question I was trying to demonstrate was that I've got a
> repo with submodules. When I push branches to most of the modules, a:
> git branch -r shows them for everyone. However, in one repo/module (I
> think it's a repo created with git --bare --shared) no one else can
> see (or pull) the remote branches and if I make a new clone of that
> repo as myself, I can't see them either. However, those branches are
> there and if I check that repo out on its own (not as a submodule of
> the main repo) I and everyone else can see those remote branches.
> 
> This is git 1.8.2.1 btw.

Thanks for your report. Unfortunately I'm not able to see much in
your attachment, could you please try to reproduce your problem with
a few shell commands and send these inline?

submodules

[shawn wilson]

I've got a git repo that just contains a bunch of small projects
(scripts, configs, etc). The idea was (is) to make it so that each
project is tracked seperately (different branches, not needing to
filter out logs for unrelated commits). And it works great for that.
The problem is that I guess I don't know how to manage it and keep
stumbling and tripping over myself here.

My issue is in trying to update the submodules, I'm getting:
 % git submodule update --init
                                                gits/kt (master ⚡)
swlap1
fatal: reference is not a tree: 98f1e9f99fca32ab3de901219eb2f600d38ab3a5
Unable to checkout '98f1e9f99fca32ab3de901219eb2f600d38ab3a5' in
submodule path 'repo_a'

Ok, so a bit of googling and I found this:
http://stackoverflow.com/questions/13425298/git-submodule-update-fatal-error-reference-is-not-a-tree
Ok, so update the repo that contains the submodules every time you
push from a sub repo? How / where do I create a hook to do this?

And then:
http://stackoverflow.com/questions/2155887/git-submodule-head-reference-is-not-a-tree-error
So, the first example didn't really work:
 % cd repo_a
                                            gits/kt (master ⚡) swlap1
 % git checkout 98f1e9
                                              kt/repo_a (master)
swlap1
error: pathspec '98f1e9' did not match any file(s) known to git.

So, at this point I realized I have pushed from two repos (maybe
others have as well) and not updated the repo of submodules. So, as
long as the issue is just with this one repo, I'm ok with loosing a
little data (not much has changed in a while and maybe I'll go and
find the blobs and merge them back in later). So I try the other
method:
 % git log --oneline -p -- repo_a
                                            gits/kt (master ⚡) swlap1
7911a1e Bump.
diff --git a/repo_a b/repo_a
index 206635e..98f1e9f 160000
--- a/repo_a
+++ b/repo_a
@@ -1 +1 @@
-Subproject commit 206635eed8b94e4133a7689b34a570e2fb0b6069
.......
 % git checkout d92f592- -- repo_a
                                            gits/kt (master ⚡) swlap1
fatal: invalid reference: d92f592-



So, (after this is solved) is there a post hook (not even sure where
such a thing would go) after pushing from a submodule repo (maybe a
concatination of commit messages or something) so that this stays in
sync and I never have to think about this again (though I think I'll
remember after winning this fight).

Re: submodules

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2002 bytes --]

On Thu, Jan 23, 2014 at 03:38:49AM -0500, shawn wilson wrote:
> My issue is in trying to update the submodules, I'm getting:
>  % git submodule update --init
>                                                 gits/kt (master ⚡)
> swlap1
> fatal: reference is not a tree: 98f1e9f99fca32ab3de901219eb2f600d38ab3a5
> Unable to checkout '98f1e9f99fca32ab3de901219eb2f600d38ab3a5' in
> submodule path 'repo_a'
> 
> Ok, so a bit of googling and I found this:
> http://stackoverflow.com/questions/13425298/git-submodule-update-fatal-error-reference-is-not-a-tree
> Ok, so update the repo that contains the submodules every time you
> push from a sub repo? How / where do I create a hook to do this?

You've got it switched.  You *did* push the superproject, but forgot
to push the new submodule commits that it references.  An easy way to
avoid this problem is to always push the superproject using:

  $ git push --recurse-submodules=on-demand …

If you no longer have access to the submodule repositories that
weren't pushed, you'll have to roll back the superproject so it
references submodule commits that were pushed.  The easiest way to do
this is probably to just cd into the submodule directory and checkout
an appropriate commit (e.g. origin/master):

  $ cd repo_a
  $ git fetch origin
  $ git checkout origin/master

That will put you on a detached HEAD, so you might want to replace the
checkout with:

  $ git checkout -B master origin/master

or some such to get a local branch.

Then cd back into the superproject and commit the submodule
(referencing the current submodule commit):

  $ cd ..
  $ git commit -m 'repo_a: Roll back to last public commit' repo_a

When you push that commit, don't forget to push everything ;)

  $ git push --recurse-submodules=on-demand

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]