Re: Making submodules easier to work with

[Finn Arne Gangstad <finnag@pvv.org>]

On Wed, Apr 30, 2008 at 06:28:00PM -0400, Avery Pennarun wrote:
> On 4/30/08, Roman Shaposhnik <rvs@sun.com> wrote:
> > On Wed, 2008-04-30 at 17:48 -0400, Avery Pennarun wrote:
> >  > It would be awesome if you could turn the fancy behaviour of this
> >  > bundle into patches to git-submodule, for example, and then have your
> >  > textmate macros call the modified git-submodule.  It might be a bit of
> >  > an uphill battle to get the patches accepted into the release, but I
> >  > think it's worth the effort, as git-submodule in its current state is
> >  > just a non-starter for my group at least.
> >
> > Doesn't the fact that the workflows around submodules tend to differ so
> >  much call for different incarnations of git-submodule? IOW, wouldn't
> >  it be ok to have an alternative to git-submodule somewhere in contrib?
> 
> This would be okay with me.  git-submodule itself doesn't seem to do
> anything very complex.  I would be happy to help contribute to such a
> thing, as my group needs it rather desperately.

Where do we want to go with submodules?

Today, submodules seem to be a "read-only" implementation of the
supermodule. By that I mean that it is (only?) suited for creating a
supermodule that consists of independently released submodules, where
all development happens in the submodules, and you sometimes update
the supermodule to refer to a new version of a submodule.

What I've tried to achieve with submodules is a bit different: I want
most development to happen in the supermodule _as if_ the submodules
were part of the supermodule. There are two reasons for not doing it
with one big module: Total size can be a bit too big, but most
importantly, some submodules are shared between different super
modules and there is a certain level of synchronizing. Does this match
your scenarios in any way?

Simplified but realistic scenario [*]:

Supermodule "crawler" has some native code, and uses submodule "os-lib".
Supermodule "indexer" has some native code, and uses submodule "os-lib".

One group works on "crawler" and releases a new crawler every 3 months.
Another group works on "indexer" and releases a new indexer every 2 weeks.

The guy who wrote "os-lib" quit 2 years ago, and no one really
maintains it. Once a year or so someone might try to unify the
"os-lib" releases (so it should be POSSIBLE, just not the norm).


What I would like: People should be able to work on "crawler" as if it
was a single git repository, in particular this means:

o Branching "crawler" means branching "os-lib"
o You can send a patch that contains changes both to "crawler" and "os-lib"
  and get it applied in a resonable way as ONE modification (and git-am
  would do the right thing)
o Merging branch a and branch b in "crawler" also merges the matching
  branches a and b in "os-lib".
o Pushing the supermodule also pushes the submodules

The branch names in "os-lib" can (and probably should) be mangled, so
they are a combination of the supermodule and the branch
name. E.g. something like crawler::branch, crawler/branch, or even
{crawler}.branch. (Is there a character that is currently not allowed in
branch names that can be used as an escape?)

Some lose implementation ideas: (code would be better, I know):

- Enable new behaviour with "git subdirectory" instead of "git submodule",
  and let "git submodule" keep the old behaviour.
- subdirectories get the following magic additions:
  - branching in parent creates {parent}.<branch> in subdirectories
  - merging branches in parent merges the corresponding {parent}.branches
    in subdirectories
  - pushing a parent also pushes the subdirectories
  - changing the parent to something that does not contain the subdirectory
    is only allowed if the subdirectory is "clean" (no uncommitted changes).


[*] In reality: there are 300+ submodules, in all states from
"actively maintained" to "last person who understood the code left 10
years ago". There are maybe 5 or 10 supermodules.

- Finn Arne