[RFC] git-submodule: multiple-level modules definition

[RFC] git-submodule: multiple-level modules definition

[Ping Yin]

1. .gitmodules supports multiple submodule level definition

This follows suggestions from Linus and and Sven Verdoolaege etc.
See "Let .git/config specify the url for submodules"
(http://article.gmane.org/gmane.comp.version-control.git/48939).

.gitmodules with with multiple level of indirection
------------------------------------------------------
[submodule "service"]
    submodule = crawler
    submodule = search
[submodule "crawler"]
    submodule = util
    submodule = imcrawter
[submodule "search"]
    submodule = util
    submodule = imsearch
[submodule "util"]                      (1)
    url = git://xyzzy/util.git
[submodule "imsearch"]
    path = search/imsearch
    url = git://xyzzy/imsearch.git
[submodule "imcrawler"]
    path = crawler/imcrawter
    url = git://xyzzy/imcrawter.git

(1): path default to submodule name
------------------------------------------------------

To simplify the case, submodule sections with submodule attribute should
have neither path attribute nor url attribute (if have, should be ignored).

2. Designate modules by logical names instead of real path

"--name|-n" options is proposed with which modules are designated by logical
names instead of real paths as following example shows.

Identical commands forms with/without "--name"
---------------------------------------------------
$ git submodule XXX util imcrawler              (1)
$ git submodule XXX -n crawler                  (2)
$ git submodule XXX util imcrawler imsearch     (3)
$ git submodule XXX -n service                  (4)
$ git submodule XXX -n crawler search           (5)

* XXX represent list, update or init, but not add
* (1) and (2) are idetical conditionally (explained below)
* (3), (4) and (5) are idetical conditionally
---------------------------------------------------

There are still minor difference between these two forms.

In the no "--name" form, the path parameter may be not the real submodule
path, and it just acts as the filter for real submodule paths.

In the "--name" form, name parameter must be the logical name, and the real
paths corresponding to the logical name may be neither a submodule path nor
even existent. How to handle such a path depends on the subcommand.

  1) status: Output 0{40} as the sha1. Doing this can remind the user to
     add the path as submodule or delete the path from .gitmodules.
  2) update: Skip that path and issue a warning
  3) init: Also init for that path

So in the example above, commands (1) and (2) are identical only when util and
imcrawler are already submodules.

3. Add submodules in batch with "git submodule add --name"

The former workflow to add submodules is adding one by one with
"git submodule add url path" which then modifies .gitmodules. However,
sometimes it may be more convenient to work in the reverse way: edit
.gitmodules first and then add submodules in batch.

Now "git submodule add --name modulename" can help us to do that. It will
find all submodules corresponding to the logical name and add them in batch
by using the path and url from .gitmodules. Of course, it will skip those
paths which have already been submodules.

4. Do we really need 'init' subcommand?

I think .gitmodules should have been bound more tightly with "git submodule".
Now it seems that 'update' or 'status' can still be performed without
.gitmodules after "git submodule init".

So why not to read submodules info from .gitmodules directly just like what
.gitignore nor .gitattributes have been dealt with? Just for editing them
more free?

>From another point, correspoingding to .gitignore we have .git/info/exclude
which just records some paths special to the local repository. But
.git/info/exclude can't replace .gitignore. But why can .git/config replace
.gitmodules which is a naive attribute of the repository?

After introducing multi-level module definition, .gitmodules becomes
more naive to the repository. And it becomes more inappropriate to put
all .gitmodules contents into .git/config since multi-level module
definition shouldn't be edited outside .gitmodules.


-- 
Ping Yin

Re: [RFC] git-submodule: multiple-level modules definition

[Lars Hjemli]

On Sun, Mar 2, 2008 at 6:11 PM, Ping Yin <pkufranky@gmail.com> wrote:
>  4. Do we really need 'init' subcommand?

Yes: .git/config tells git which submodule(s) the user is interested
in and where to fetch updates from for these submodules, while
.gitmodules is used to map a submodule _name_ to a submodule _path_
and also to provide a url where submodule updates _can_ be fetched.

So the actual use of submodules is strictly a local configuration
issue and as such it needs to be kept in .git/config. And to help the
user populate .git/config with submodule information we have 'git
submodule init'.

--
larsh

Re: [RFC] git-submodule: multiple-level modules definition

[Ping Yin]

On Mon, Mar 3, 2008 at 2:18 AM, Lars Hjemli <hjemli@gmail.com> wrote:
> On Sun, Mar 2, 2008 at 6:11 PM, Ping Yin <pkufranky@gmail.com> wrote:
>  >  4. Do we really need 'init' subcommand?
>
>  Yes: .git/config tells git which submodule(s) the user is interested
>  in and where to fetch updates from for these submodules, while
>  .gitmodules is used to map a submodule _name_ to a submodule _path_
>  and also to provide a url where submodule updates _can_ be fetched.
>
>  So the actual use of submodules is strictly a local configuration
>  issue and as such it needs to be kept in .git/config. And to help the
>  user populate .git/config with submodule information we have 'git
>  submodule init'.
Is it always a local configuration issue? In a project with hundreds
of submodule, will anybody modify the submodule url?
I think .gitmodules can be the default repository configuration, and
when we have local configuration issue, we can override the
configuration in .git/config with the help of git init. But why always
use git init even when we don't have the local configuration
requirement?
>
>  --
>  larsh
>



-- 
Ping Yin

Re: [RFC] git-submodule: multiple-level modules definition

[Lars Hjemli]

On Mon, Mar 3, 2008 at 2:47 AM, Ping Yin <pkufranky@gmail.com> wrote:
>  But why always
>  use git init even when we don't have the local configuration
>  requirement?

We need a way to tell git which of those submodules we are interested
in, i.e. which submodules to clone/fetch during 'git submodule
update'.

--
larsh

Re: [RFC] git-submodule: multiple-level modules definition

[Ping Yin]

On Mon, Mar 3, 2008 at 2:57 PM, Lars Hjemli <hjemli@gmail.com> wrote:
> On Mon, Mar 3, 2008 at 2:47 AM, Ping Yin <pkufranky@gmail.com> wrote:
>  >  But why always
>  >  use git init even when we don't have the local configuration
>  >  requirement?
>
>  We need a way to tell git which of those submodules we are interested
>  in, i.e. which submodules to clone/fetch during 'git submodule
>  update'.
>
Now that's easy with the multiple level module definition. We can now
group our modules by hierachy and designate modules we are interested
by logical names such as crawler or search as my proposal has
mentioned.

And if there are not a logical module name satisfying our requirement,
we can define one by ourself in .git/config (if it's a special or
temporary requirement) or  in .gitmodules (if it's a common
requirement).

Now .git/config and .gitmodules will play the same as
.git/info/exclude and .gitignore.
>  --
>  larsh
>



-- 
Ping Yin

Re: [RFC] git-submodule: multiple-level modules definition

[Lars Hjemli]

On Mon, Mar 3, 2008 at 12:02 PM, Ping Yin <pkufranky@gmail.com> wrote:
>
> On Mon, Mar 3, 2008 at 2:57 PM, Lars Hjemli <hjemli@gmail.com> wrote:
>  > On Mon, Mar 3, 2008 at 2:47 AM, Ping Yin <pkufranky@gmail.com> wrote:
>  >  >  But why always
>  >  >  use git init even when we don't have the local configuration
>  >  >  requirement?
>  >
>  >  We need a way to tell git which of those submodules we are interested
>  >  in, i.e. which submodules to clone/fetch during 'git submodule
>  >  update'.
>  >
>  Now that's easy with the multiple level module definition. We can now
>  group our modules by hierachy and designate modules we are interested
>  by logical names such as crawler or search as my proposal has
>  mentioned.

Sorry, you've lost me here. How is 'git submodule update' going to
avoid cloning/fetching all submodules with your proposal?

PS: the supposed work-flow is an initial 'git submodule init foo bar
baz' (i.e. register the submodules you're interested in checking out)
followed by an occasional 'git submodule update' (i.e. update all the
submodules you are interested in).

--
lh

Re: [RFC] git-submodule: multiple-level modules definition

[Ping Yin]

On Mon, Mar 3, 2008 at 7:51 PM, Lars Hjemli <hjemli@gmail.com> wrote:
> On Mon, Mar 3, 2008 at 12:02 PM, Ping Yin <pkufranky@gmail.com> wrote:
>  >
>  > On Mon, Mar 3, 2008 at 2:57 PM, Lars Hjemli <hjemli@gmail.com> wrote:
>  >  > On Mon, Mar 3, 2008 at 2:47 AM, Ping Yin <pkufranky@gmail.com> wrote:
>  >  >  >  But why always
>  >  >  >  use git init even when we don't have the local configuration
>  >  >  >  requirement?
>  >  >
>  >  >  We need a way to tell git which of those submodules we are interested
>  >  >  in, i.e. which submodules to clone/fetch during 'git submodule
>  >  >  update'.
>  >  >
>  >  Now that's easy with the multiple level module definition. We can now
>  >  group our modules by hierachy and designate modules we are interested
>  >  by logical names such as crawler or search as my proposal has
>  >  mentioned.
>
>  Sorry, you've lost me here. How is 'git submodule update' going to
>  avoid cloning/fetching all submodules with your proposal?
>
>  PS: the supposed work-flow is an initial 'git submodule init foo bar
>  baz' (i.e. register the submodules you're interested in checking out)
>  followed by an occasional 'git submodule update' (i.e. update all the
>  submodules you are interested in).
>

See my first email. In following .gitmodules example, we can use "git
submodule update service" to update all sub-submodules of submodule
"service" (util, imsearch, imcrawler). We don't have to do "git init"
first.

Without the multiple level module definition, "git init" is good idea,
since every time we want to 'git update', we needn't give so many
module paths. However, if we introduce multiple level module
definition, we can define the module groups we are interested such as
crawler, search and the toppest group service. So we just need to type
one or few module names which makes "git init" not so useful.

And it's more flexible by using module name: at one time we can update
"cralwer" module group, and at another time we can update "search"
module group.


.gitmodules example
------------------------------------------------------
[submodule "service"]
   submodule = crawler
   submodule = search
[submodule "crawler"]
   submodule = util
   submodule = imcrawter
[submodule "search"]
   submodule = util
   submodule = imsearch
[submodule "util"]                      (1)
   url = git://xyzzy/util.git
[submodule "imsearch"]
   path = search/imsearch
   url = git://xyzzy/imsearch.git
[submodule "imcrawler"]
   path = crawler/imcrawter
   url = git://xyzzy/imcrawter.git
------------------------------------------------------
>  --
>  lh
>



-- 
Ping Yin

Re: [RFC] git-submodule: multiple-level modules definition

[Ping Yin]

On Mon, Mar 3, 2008 at 1:11 AM, Ping Yin <pkufranky@gmail.com> wrote:
> 1. .gitmodules supports multiple submodule level definition
>
>  This follows suggestions from Linus and and Sven Verdoolaege etc.
>  See "Let .git/config specify the url for submodules"
>  (http://article.gmane.org/gmane.comp.version-control.git/48939).
>
>  .gitmodules with with multiple level of indirection
>  ------------------------------------------------------
>  [submodule "service"]
>     submodule = crawler
>     submodule = search
>  [submodule "crawler"]
>     submodule = util
>     submodule = imcrawter
>  [submodule "search"]
>     submodule = util
>     submodule = imsearch
>  [submodule "util"]                      (1)
>     url = git://xyzzy/util.git
>  [submodule "imsearch"]
>     path = search/imsearch
>     url = git://xyzzy/imsearch.git
>  [submodule "imcrawler"]
>     path = crawler/imcrawter
>     url = git://xyzzy/imcrawter.git
>
>  (1): path default to submodule name
>  ------------------------------------------------------
>
>  To simplify the case, submodule sections with submodule attribute should
>  have neither path attribute nor url attribute (if have, should be ignored).
>
>  2. Designate modules by logical names instead of real path
>
>  "--name|-n" options is proposed with which modules are designated by logical
>  names instead of real paths as following example shows.
>
>  Identical commands forms with/without "--name"
>  ---------------------------------------------------
>  $ git submodule XXX util imcrawler              (1)
>  $ git submodule XXX -n crawler                  (2)
>  $ git submodule XXX util imcrawler imsearch     (3)
>  $ git submodule XXX -n service                  (4)
>  $ git submodule XXX -n crawler search           (5)
>
>  * XXX represent list, update or init, but not add
>  * (1) and (2) are idetical conditionally (explained below)
>  * (3), (4) and (5) are idetical conditionally
>  ---------------------------------------------------
>

Maybe i should introduce more about the background.

In my company we try to keep all the codes in a unique big environment
which consists of many sub projects (more than 30). These subprojects
are hierachily organized. There are different roles which need touch
different part of these sub projects. For example, crawler deveoper
should touch crawler-related projects, and search developer need touch
search-related projects. There are also some common modules which are
used by most of these roles. There are also many other roles.

So for easy use, we need define one logical name for each kind of role
and this name should correspond to all the role-related sub projects.
Filtering submodules by path is not so useful for us because
role-related projects may live in many different directories.

>From the management view of company,  all the roles should use the
same big environment. And we don't want to use different branch for
different roles. So all roles will get the same .gitmodules file. They
are not much familiar with git and can't afford to type git init path1
... pathn each time when they want to build their develop environment.
It's more friendly to them if they only need to remember a logical
name such as crawler or search.

And some people have multiple roles. Sometime they want to work on
search and sometimes on crawler. So logical name is very convenient to
them.

As a conclusion, my main point is
1. Multiple level module definition is very useful since it can help
us build hierachy of modules
2. Designating modules by logical name which corresponds to a group of
role-related modules is usefull since it can help role-based
development much.



-- 
Ping Yin