Submodules use case: web development based on modular CMS

Submodules use case: web development based on modular CMS

[Jakub Narebski]

A bit of time ago I have stumbled upon the following blog entry 
(question): "Agaric wants version control that lets Drupal core and 
contrib replace entire directories within our checkouts"[1]
  http://tinyurl.com/yv3jp4

Here the author want to use version control system for customized 
version of CMS system, namely:
 * Drupal core
 * selected Drupal contributed modules ("inside" Drupal core)
 * own work (theme, custom modules) (also "inside" Drupal core)
And of course he wants to have development branches: dev, test and live, 
and want to always checkout correct versions of Drupal, Drupal modules 
and own work.

Somebody (in comments) proposed using Subversion externals; I have 
proposed using Git submodules (subproject) support.  Nevertheless while 
I have at one time manage PHPNuke portal using CVS and vendor branches, 
and even had half-written automatic fast checkout/fast export script, 
I haven't worked with submodules in Git, nor with using Git to manage 
web site.

Comments (here, and as comments to mentioned blog article) about how 
such thing should be done with Git very appreciated. (And for example 
how Git submodules differ from svn:externals and why git way is 
better ;-)).


P.S. BTW. Drupal is considering moving from CVS to some other SCM, see
http://groups.drupal.org/revision-control-systems .  Perhaps Git would 
be the choice?  Is anybody here active in Drupal development?  Is for 
example their Version Control API too CVS-centric (single file history 
centric)?

[1]http://agaricdesign.com/note/agaric-wants-version-control-lets-drupal-core-and-contrib-replace-entire-directories-within-our-checkouts
-- 
Jakub Narebski
Poland

Re: Submodules use case: web development based on modular CMS

[Martin Langhoff]

On Jan 27, 2008 1:39 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> A bit of time ago I have stumbled upon the following blog entry
> (question): "Agaric wants version control that lets Drupal core and
> contrib replace entire directories within our checkouts"[1]
>   http://tinyurl.com/yv3jp4

While it is possible to use submodules, I tend to think that you can
just use your own repo / branch "off" the main project, just like
unofficial kernel modules do.

It does have a number of advantages -- including being explicit wrt to
what version of the core project the module is developed against.
Drupal and similar projects (like Moodle) do have a module API, but
it's not *that* stable -- in spite of the best intentions, those APIs
get subtle changes all the time. You should have your "release" script
package only the module subdirectories -- and perhaps any delta in the
core (of Drupal) that needs to be applied. In many "contrib" projects
I've seen module files and little patch files that you are supposed to
apply - and maintenance of those is a pain. It makes more sense to use
git over the whole tree.

So it's a tradeoff -- IMHO, at first blush it looks more "natural" to
use submodules, but as things progress over time I think it forces a
lot of additional and ill-fitted work like maintaining patches, or
adding tags that indicate "developed against Drupal v1.2.3". And the
day the contrib module becomes official, "merging it in" is a bit of a
mess if you want to preserve history. So in the long haul, it makes a
lot more sense to use the "branch off the full project" approach that
has served the kernel modules so well.

In that sense, I think that gitk's tree getting merged in a
subdirectory is good as an example of what git can do, but a bit
pointless as gitk depend quite tightly on the behaviour of git
commands. So distributing gitk separately would be a big pain as each
gitk version is usable against a narrow set of git revisions. Luckily,
it gets merged into git and that's how it gets distributed.

My .2c anyway ;-)


m

Re: Submodules use case: web development based on modular CMS

[Jakub Narebski]

I CC'ed drupal mainling list...

On Sun, 27 Jan 2008, Martin Langhoff wrote:
> On Jan 27, 2008 1:39 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>>
>> A bit of time ago I have stumbled upon the following blog entry
>> (question): "Agaric wants version control that lets Drupal core and
>> contrib replace entire directories within our checkouts"[1]
>>   http://tinyurl.com/yv3jp4
> 
> While it is possible to use submodules, I tend to think that you can
> just use your own repo / branch "off" the main project, just like
> unofficial kernel modules do.

I think it is why they currently do, and they want to change it...

<blockquote href="http://tinyurl.com/yv3jp4">
  Right now we have each of these three areas checked out independently,
  and while they do happily reside inside one another, there's some
  inconvenience and risk of not keeping everything in synch.

  The solution we're looking at now is shell scripts but if there's
  a way for a version control system to just understand this need and
  accept it, that would be best.
</quote>

...but I guess they use CVS just like Drupal does, which might make it
painful.

> It does have a number of advantages -- including being explicit wrt to
> what version of the core project the module is developed against.
> Drupal and similar projects (like Moodle) do have a module API, but
> it's not *that* stable -- in spite of the best intentions, those APIs
> get subtle changes all the time. You should have your "release" script
> package only the module subdirectories -- and perhaps any delta in the
> core (of Drupal) that needs to be applied. In many "contrib" projects
> I've seen module files and little patch files that you are supposed to
> apply - and maintenance of those is a pain. It makes more sense to use
> git over the whole tree.
> 
> So it's a tradeoff -- IMHO, at first blush it looks more "natural" to
> use submodules, but as things progress over time I think it forces a
> lot of additional and ill-fitted work like maintaining patches, or
> adding tags that indicate "developed against Drupal v1.2.3". And the
> day the contrib module becomes official, "merging it in" is a bit of a
> mess if you want to preserve history. So in the long haul, it makes a
> lot more sense to use the "branch off the full project" approach that
> has served the kernel modules so well.

I think that choice between developing "in core" / on branch like Linux 
kernel modules and subsystems vs using submodules support like msysGit, 
cgit and (perhaps) in the future git-gui and gitk, depends on a few 
things:
 1. How tightly coupled are modules with core / how independent
    development of modules is
 2. How tightly coupled / how independent modules are from each other,
    both from the development point of view, but also from testing point
    of view
 3. How much work has to be done to add / enable module

How coupled modules are with the core depends among others on the 
stability of modular API.  The dependency of modules on core 
development might be measured by [relative] number of commits which 
change core and module, or which change module responding to change
in core (bugfixes and workaround for core bugs excluded).  And of course 
if module development _is_ done independently.

If modules are not developed in isolation (e.g. code movement), or 
cannot be tested in isolation (unit tests for modules alone, whole 
tests for core + single module) then of course it doesn't make much 
sense to use submodules support.  (For example it would be hard to test 
for example networking module without filesystem module, for Linux 
kernel modules.)

If you have to run some install script to install / start using modules 
(for example phpBB modules are custom files + specialized patch-like 
file, which allows modifying not only files but also databases), then 
that's of no avail to use submodules because you have to use [custom] 
scripts to update nevertheless...

> In that sense, I think that gitk's tree getting merged in a
> subdirectory is good as an example of what git can do, but a bit
> pointless as gitk depend quite tightly on the behaviour of git
> commands. So distributing gitk separately would be a big pain as each
> gitk version is usable against a narrow set of git revisions. Luckily,
> it gets merged into git and that's how it gets distributed.

Errr... AFAICT it is not true. It is true that gitk and git-gui require 
sufficiently new git version to work, but IIRC both have feature 
autodetection, which allows to use gitk and git-gui with older git (and 
of course the reverse: old gitk or git-gui with new git). I have used 
gitk from 'master' occassionaly, while having as default some older git 
version installed from RPM (released version).

gitk and git-gui development is managed independently from git, and can 
be done using only gitk and git-gui sources.  So using submodules is a 
good idea.  Additionally this allows to not checkout gitk and git-gui 
for example if you don't have Tcl/Tk installed or don't use windowed 
environment.

gitweb could also be managed as module... but currently there is no 
maintainer who would manage gitweb.git repository...


On the other hand having gitk, git-gui, gitweb in git proper allow to 
use just introduced features, and not only features available in last 
released version.


BTW. if some tool is tightly coupled with git, it can use git.git 
repository (or modification of thereof) as submodule, as shown in the 
cases of msysGit and cgit.

-- 
Jakub Narebski
Poland

Re: Submodules use case: web development based on modular CMS

[Jakub Narebski]

On Sun, 27 Jan 2008, Jakub Narebski wrote:

> Somebody (in comments) proposed using Subversion externals; I have 
> proposed using Git submodules (subproject) support. 

BTW. Mercurial supports something similar by so called (unofficial) 
"forest extension":
  http://www.selenic.com/mercurial/wiki/index.cgi/ForestExtension

If I understand it corectly the mechanism follow idea of manifest-like 
file Mercurial uses, and is similar to one of the test implementations 
of submodule support before we had current "in core" version.

I wonder if Bazaar-NG has similar extension / plugin...


It would be nice to have comparison of submodule / subproject support in 
different version control systems...
-- 
Jakub Narebski
Poland