How to manage multiple repos using submodules?

How to manage multiple repos using submodules?

[Andrew Wong]

It seems like submodule isn't meant for this, but many people seems to 
use submodule to link many smaller repos together. With this setup, I 
imagine whenever someone pushed a small repo, they're /supposed/ to push 
the big repo as well. This way, if I simply update the big repo and do a 
"git status", git will tell me that which of the smaller repos are out 
of date. However, this is only reliable if everyone remembers to push 
the big repo. If someone pushed a smaller repo, but forgot to push the 
big repo, then I won't be aware that some of the smaller repos are 
out-of-date until I push.

I suppose one way is to do a check/auto-update with pre/post commit 
script to enforce that the big repo is always up-to-date. Another way is 
to use the "submodule foreach" to do a fetch and status on smaller ones, 
but this doesn't seem very elegant.

So, I'm wondering how do people who use submodules this way manage all 
their repos? Or how to (reliably) get a good sense of the state of the 
small repos? Or maybe I shouldn't be using submodules this way at all?

Re: How to manage multiple repos using submodules?

[Jonathan Nieder]

Hi Andrew,

Andrew Wong wrote:

> It seems like submodule isn't meant for this, but many people seems
> to use submodule to link many smaller repos together. With this
> setup, I imagine whenever someone pushed a small repo, they're
> /supposed/ to push the big repo as well. This way, if I simply
> update the big repo and do a "git status", git will tell me that
> which of the smaller repos are out of date.

Yep, if you want to keep track of the state of a bunch of repos over
time, submodules are not so bad[*].  In practice, often one instead
wants to keep a bunch of repos up-to-date, and all this meta-history
tracking is overkill.

I'd suggest using the mr tool.  Some projects (e.g., the
debian-installer project) are using it to help people keep all their
repos up to date.

http://kitenet.net/~joey/code/mr/

Hope that helps,
Jonathan

Re: How to manage multiple repos using submodules?

[Jonathan Nieder]

Hi,

Jonathan Nieder wrote:

> Yep, if you want to keep track of the state of a bunch of repos over
> time, submodules are not so bad[*].

A kind person pointed out that I left out a footnote.  I think all I
had been planning to say is that, roughly speaking, submodules are
about[1] saying that a specific commit is known to work well with the
rest of the code.  A supermodule like the one discussed in [2] is only
likely to be useful if you are interested in what historical
combinations of repositories were published and meant to work well
together.

Ciao,
Jonathan

[1] e.g., http://thread.gmane.org/gmane.comp.version-control.git/27803/focus=27830
[2] http://lists.x.org/archives/xorg-devel/2009-September/001966.html

Re: How to manage multiple repos using submodules?

[Andrew Wong]

Thanks for the mail archives! They were a very good read.

The smaller projects I have are fairly independent, so there isn't a 
situation where a commit works well with the other commits. I just 
wanted some ways to split each project out to its own repo. So that when 
I want to do some git operations on one project, I don't have to worry 
about the other projects. While submodules isn't an ideal solution, it 
seems to be the closest. Maybe what I need from submodules is a way for 
the super-repo to not record the commit of the sub-repos. i.e. Just use 
the head of a branch. But if that's the case, maybe it's out of the 
scope of a SCM, since I'm not really tracking a history anymore.

I haven't tried it yet, but the mr tool you mentioned seems interesting 
too. I'll check it out.

Thanks!
Andrew


On 11-04-17 2:48 AM, Jonathan Nieder wrote:
> Hi,
>
> Jonathan Nieder wrote:
>
>> Yep, if you want to keep track of the state of a bunch of repos over
>> time, submodules are not so bad[*].
> A kind person pointed out that I left out a footnote.  I think all I
> had been planning to say is that, roughly speaking, submodules are
> about[1] saying that a specific commit is known to work well with the
> rest of the code.  A supermodule like the one discussed in [2] is only
> likely to be useful if you are interested in what historical
> combinations of repositories were published and meant to work well
> together.
>
> Ciao,
> Jonathan
>
> [1] e.g., http://thread.gmane.org/gmane.comp.version-control.git/27803/focus=27830
> [2] http://lists.x.org/archives/xorg-devel/2009-September/001966.html
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to manage multiple repos using submodules?

[Phil Hord]

Thanks, Jonathan.

On 04/16/2011 02:20 PM, Jonathan Nieder wrote:
> Hi Andrew,
>
> Andrew Wong wrote:
>
>> It seems like submodule isn't meant for this, but many people seems
>> to use submodule to link many smaller repos together. With this
>> setup, I imagine whenever someone pushed a small repo, they're
>> /supposed/ to push the big repo as well. This way, if I simply
>> update the big repo and do a "git status", git will tell me that
>> which of the smaller repos are out of date.
> Yep, if you want to keep track of the state of a bunch of repos over
> time, submodules are not so bad[*].  In practice, often one instead
> wants to keep a bunch of repos up-to-date, and all this meta-history
> tracking is overkill.

I had been thinking about this a lot lately.  We are using submodules
somewhat extensively to manage a project with lots of common code
feeding onto 12 different platforms (various OSes, compilers and SOCs). 
It all feels quiet unwieldy but, at the same time, necessary.

I looked briefly at the android repo tool, but it's not just where I
want it to be either. Also, it loses history (by design, though that
seems to be a mistake they intend to correct someday).

But all this thinking got me closer to understanding that there are two
competing needs here, as you both pointed out:
  1. Point-in-time history using SHA-1s (the git model)
  2. Branch following (the 'repo', mr, gitslave or svn model)

When I am looking for a bug introduced in the past, I want to have the
first model.  When I am eager to pull in bug fixes, new features, or a
different meta-branch from other developers (and possibly other
projects), I want the second model.

Submodules make it difficult to manage both at the same time.   It
requires extra steps to do what it should be doing by default (acting
like git, tracking project point-in-time history).  I understand why it
is like this, but empathy doesn't ease the pain.

But I really do want both models.  I want point-in-time history, and I
want semi-automatic branch following.  Maybe something like this:

1. Easily record point-in-time history
   # Create and switch to a branch on superproject and all submodules
   git checkout -b topic-foo --recursive

   submodule A/topic-foo:  hack, hack, hack
   submodule B/topic-foo:  hack, hack, hack

   # Commit and push changes on each submodule
   git commit --recursive
   git push origin HEAD:ph/foo --recursive

2. Ignore the gitlinks and just checkout master everywhere
   git checkout master --recursive
  

I think I can do both with just a few tweaks of the submodule
machinery.  I haven't had time to spelunk there much, but I see others
are working to advance submodules in somewhat similar directions. 
(Thanks for that; it is much appreciated.)

Have these dueling modes been discussed into the dirt somewhere
already?  I find lots of discussions about other tools to perform model
2 instead of model 1, but I don't see much discussion about the
simultaneous need for both.   Maybe I am overthinking it.  Or maybe my
particular code model is unusual.

> I'd suggest using the mr tool.  Some projects (e.g., the
> debian-installer project) are using it to help people keep all their
> repos up to date.
>
> http://kitenet.net/~joey/code/mr/

Thanks.  I hadn't seen that one before.  I'll take a look.

Phil