[RFC] Referring to a submodule state recorded in a supermodule from within the submodule

[RFC] Referring to a submodule state recorded in a supermodule from within the submodule

[Johan Herland]

Hi,

I have a stand-alone project, "foo", that I work on myself. The "foo" 
project is included as a submodule in two other projects, "bar" 
and "baz", that I don't have any direct affiliation with.

Semi-regularly, I like to keep tabs on bar and baz, to see what versions 
of foo they are using, what changes they have made to foo, and if there 
are things I could pick up from them, or maybe even things they could 
learn from eachother.

Doing this currently is quite tedious:
1. Clone/Fetch bar and initialize/update its foo submodule
2. Clone/Fetch baz and initialize/update its foo submodule
3. Set up remotes bar_foo and baz_foo in my main foo repo,
   pointing to bar/foo and baz/foo, respectively. Fetch.
4. Create tags bar_foo_current and baz_foo_current pointing
   to the foo SHA1 sum recorded in baz and baz, respectively.
5. Start comparing bar_foo_current and baz_foo_current to
   eachother, and to my own master branch.


Now, bring this into a larger company setting. There are many (~100) 
different foo-type projects owned by (~50) different developers. These 
projects are included as submodules by several (~20) different 
bar/baz-type projects. In addition, each bar/baz-type project has 
multiple (~3) branches using various versions of the foo-type 
submodules.

Finally, throw in questions like "What are the differences in submodule 
FOO between branch X of BAR, and branch Y of BAZ?", for random values 
of FOO, X, BAR, Y, and BAZ. It is apparent that the above approach just 
doesn't cut it.

Ironically, these questions are fairly easily answered by our current 
CVS setup (which applies the supermodule tags/branches to the entire 
source tree (including submodules)):
1. cd FOO
2. cvs diff -r BAR_X -r BAZ_Y

What I'd like, is some way to refer to the state of a repo as specified 
by the appropriate submodule blob in a superproject commit. Ideally 
this should be done remotely as well, so that I don't have to clone the 
entire superproject just to get at the appropriate submodule blob.


Crude Proposal:

Define a new "git submodule" subcommand that takes three arguments:
   <superURL> <tree-ish> <submoduleName>
The command does the following steps:
1. Locate git repo at <superURL>
2. Resolve <tree-ish> to the tree object within the git repo in #1
3. Locate .gitmodules within the tree object in #2
4. Lookup <submoduleName> in .gitmodules to find its path and URL
5. Locate the submodule blob from the path in #4 within the tree from #2
6. Record the object name (SHA1) stored in the submodule blob from #4
7. Fetch the object name (#7) from the submodule repo (its URL was
   found in #4) into the local git repo
8. Store a reference to the fetched object


Given the above command, the tedious steps described at the top can be 
reduced to the following (which is reasonably close to the equivalent 
CVS commands):
1. git submodule magic-command url/to/bar master foo
2. git submodule magic-command url/to/baz master foo
3. Start comparing the fetched refs to eachother, and
   to my own master branch.

If this sounds reasonable, I'd be happy to start coding the above 
proposal.


Have fun! :)

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: [RFC] Referring to a submodule state recorded in a supermodule from within the submodule

[Junio C Hamano]

Johan Herland <johan@herland.net> writes:

> I have a stand-alone project, "foo", that I work on myself. The "foo" 
> project is included as a submodule in two other projects, "bar" 
> and "baz", that I don't have any direct affiliation with.
>
> Semi-regularly, I like to keep tabs on bar and baz, to see what versions 
> of foo they are using, what changes they have made to foo, and if there 
> are things I could pick up from them, or maybe even things they could 
> learn from eachother.
>
> Doing this currently is quite tedious:
> 1. Clone/Fetch bar and initialize/update its foo submodule
> 2. Clone/Fetch baz and initialize/update its foo submodule

If I am reading you right and you are only interested in the part "foo" in
these projects, there is something wrong with the setup of "bar" and "baz".

The submodule mechanism is designed to bind an independent project on its
own as a subdirectory of another project.  It seems to me that the problem
is that "bar" and "baz" projects do not give direct access to clone "foo"
part of them for you or other people.

Re: [RFC] Referring to a submodule state recorded in a supermodule from within the submodule

[Johan Herland]

On Wednesday 05 November 2008, Junio C Hamano wrote:
> Johan Herland <johan@herland.net> writes:
> > I have a stand-alone project, "foo", that I work on myself. The
> > "foo" project is included as a submodule in two other projects,
> > "bar" and "baz", that I don't have any direct affiliation with.
> >
> > Semi-regularly, I like to keep tabs on bar and baz, to see what
> > versions of foo they are using, what changes they have made to foo,
> > and if there are things I could pick up from them, or maybe even
> > things they could learn from eachother.
> >
> > Doing this currently is quite tedious:
> > 1. Clone/Fetch bar and initialize/update its foo submodule
> > 2. Clone/Fetch baz and initialize/update its foo submodule
>
> If I am reading you right and you are only interested in the part
> "foo" in these projects, there is something wrong with the setup of
> "bar" and "baz".
>
> The submodule mechanism is designed to bind an independent project on
> its own as a subdirectory of another project.  It seems to me that
> the problem is that "bar" and "baz" projects do not give direct
> access to clone "foo" part of them for you or other people.

No, they do. I can clone foo directly from bar/baz's server. The problem 
is that I cannot quickly get at which revision of foo is used by 
bar/baz without tedious cloning and interacting with the superrepos.

I basically want to quickly answer questions like "What are the 
differences between bar's foo and baz's foo (and my own foo)?"


...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: [RFC] Referring to a submodule state recorded in a supermodule from within the submodule

[Daniel Barkalow]

On Wed, 5 Nov 2008, Johan Herland wrote:

> Hi,
> 
> I have a stand-alone project, "foo", that I work on myself. The "foo" 
> project is included as a submodule in two other projects, "bar" 
> and "baz", that I don't have any direct affiliation with.
> 
> Semi-regularly, I like to keep tabs on bar and baz, to see what versions 
> of foo they are using, what changes they have made to foo, and if there 
> are things I could pick up from them, or maybe even things they could 
> learn from eachother.
> 
> Doing this currently is quite tedious:
> 1. Clone/Fetch bar and initialize/update its foo submodule
> 2. Clone/Fetch baz and initialize/update its foo submodule
> 3. Set up remotes bar_foo and baz_foo in my main foo repo,
>    pointing to bar/foo and baz/foo, respectively. Fetch.
> 4. Create tags bar_foo_current and baz_foo_current pointing
>    to the foo SHA1 sum recorded in baz and baz, respectively.
> 5. Start comparing bar_foo_current and baz_foo_current to
>    eachother, and to my own master branch.

Set up remotes in your foo:

[remote "bar"]
	url = ...
	fetch = refs/heads/*:refs/remotes/bar/*
[remote "bar-foo"]
	url = ...
	fetch = refs/heads/*:refs/remotes/in-bar/*

You should be able to do "git fetch bar" (you really don't want to merge 
this, but git is fine with unrelated histories in the same repo...);
"git fetch bar-foo".

Then I'd expect "git log master..bar/master:foo" to work [*1*].

bar/master is the commit in the superproject, "foo" is the path in its 
tree that reaches the submodule, so "bar/master:foo" is whatever's there, 
which turns out to be a commit, which is the commit you're interested in.

I think the only annoying things are that you need to find out from 
bar/master:.gitmodules what to fetch as the branch that's related to your 
local project, and you need to do two fetches each time, and you have to 
download the whole superproject state in order to find out one reference 
from it. (But this also lets you look at the history of which versions 
they incorporated over time, so you can see "they adopted something at one 
point, rolled back for a while, cherry-picked a change, and then updated 
to a version with the feature fixed.")

[*1*] This may not actually work; I don't know if we give up on 
interpreting a object name when we're looking for a commit and start to 
see a path. I'm pretty sure that git will not understand 
"refs/remotes/bar/master:path/to/foo:Makefile" (which could name the 
Makefile in the root directory of the commit in the path/to/foo location 
in the current value of bar/master, but will actually look for a 
"foo:Makefile" filename; it probably ought to work with a '/' before the 
second ':', though, since that is unambiguous)

	-Daniel
*This .sig left intentionally blank*