Partial checkouts / submodules

Partial checkouts / submodules

[Finn Arne Gangstad]

My use case it this: We have some huge projects (let's call them
supermodules) that are the only products we really release. Any change
going into any of the submodules go in solely to modify the
superproject, the submodules are not released on their own.

We cannot keep the supermodule with all its submodules in one git
repository for two reasons: Size & sharing. A 6GB+ repository is too
big to handle gracefully, and there are multiple superprojects sharing
some of the submodules. Our supermodules typically contains 50-250
submodules. Usually it is sufficient to look at just a few of these
submodules at the same time.

I looked into the current git submodules to see if they support what I
think we need, but it seems like they do not really cut it (If I'm
wrong about this, please educate me).

What I want is this: 

Somewhere the following modules all exist:

supermodule/
   submodule1
   submodule2
   submodule3
    ...
   submodule200

You pull the supermodule, and initialize random collection of
submodules, e.g. locally you have:

supermodule/
   submodule13
   submodule71
   submodule102

Now I want this to behave as if it was a partial checkout of
"supermodule" - i.e. I want _all_ operations in any of the submodules
to act as if they happened in all the submodules (as if supermodule
was a single repository containing all the submodules directly).

If I do a change in submodule13, another change in submodule71 and yet
another change in submodule102, I want to be able to commit them all
as ONE commit (obviously it will be 4 commits, 1 in each submodule and
one in the supermodule, but anyone looking at this in the context of
this supermodule should see it as one commit).

If I pull supermodule, I get updates to supermodule, submodule13,
submodule71 and submodule102, but nothing else.

If I make a branch on submodule71, the branch is made in all submodules &
the supermodule.

With this setup it should be possible to handle supermodule as a
normal module with branches for each feature/topic/bugfix, and those
features being merged into other branches in a reasonable way. Does
something like this look doable?

- Finn Arne

Re: Partial checkouts / submodules

[Sven Verdoolaege]

On Tue, Nov 20, 2007 at 04:59:22PM +0100, Finn Arne Gangstad wrote:
> I looked into the current git submodules to see if they support what I
> think we need, but it seems like they do not really cut it (If I'm
> wrong about this, please educate me).

Maybe you could explain why you think they don't cut it.

> You pull the supermodule, and initialize random collection of
> submodules, e.g. locally you have:
> 
> supermodule/
>    submodule13
>    submodule71
>    submodule102

Just "submodule init" and "submodule update" these submodules and
it looks like you would get what you want...

> If I make a branch on submodule71, the branch is made in all submodules &
> the supermodule.

... except this one.
It's not clear why you would even want this.

skimo

Re: Partial checkouts / submodules

[Daniel Barkalow]

On Tue, 20 Nov 2007, Finn Arne Gangstad wrote:

> My use case it this: We have some huge projects (let's call them
> supermodules) that are the only products we really release. Any change
> going into any of the submodules go in solely to modify the
> superproject, the submodules are not released on their own.
> 
> We cannot keep the supermodule with all its submodules in one git
> repository for two reasons: Size & sharing. A 6GB+ repository is too
> big to handle gracefully, and there are multiple superprojects sharing
> some of the submodules. Our supermodules typically contains 50-250
> submodules. Usually it is sufficient to look at just a few of these
> submodules at the same time.
> 
> I looked into the current git submodules to see if they support what I
> think we need, but it seems like they do not really cut it (If I'm
> wrong about this, please educate me).
> 
> What I want is this: 
> 
> Somewhere the following modules all exist:
> 
> supermodule/
>    submodule1
>    submodule2
>    submodule3
>     ...
>    submodule200
> 
> You pull the supermodule, and initialize random collection of
> submodules, e.g. locally you have:
> 
> supermodule/
>    submodule13
>    submodule71
>    submodule102
> 
> Now I want this to behave as if it was a partial checkout of
> "supermodule" - i.e. I want _all_ operations in any of the submodules
> to act as if they happened in all the submodules (as if supermodule
> was a single repository containing all the submodules directly).
> 
> If I do a change in submodule13, another change in submodule71 and yet
> another change in submodule102, I want to be able to commit them all
> as ONE commit (obviously it will be 4 commits, 1 in each submodule and
> one in the supermodule, but anyone looking at this in the context of
> this supermodule should see it as one commit).

This has theoretical problems: it's going to be practically impossible, in 
most cases, to write a commit message that describes changes in three 
submodules (which are sometimes used in the context of a different 
supermodule) as well as the supermodule.

On the other hand, the supermodule commit is the one that's relevant in 
the context of the supermodule, and that commit's message should describe 
the set of changes as a whole.

I think it should be possible and sufficient to have "git commit" able to 
recursively start a "git commit" in submodules (where you'd write a 
separate message suitable for exposure to other supermodules), so you'd 
have to write 4 messages but only type one command line.

> If I pull supermodule, I get updates to supermodule, submodule13,
> submodule71 and submodule102, but nothing else.

This should work. Of course, if other people have made changes to parts of 
the supermodule that you haven't checked out, your supermodule index will 
reflect those changes, so that supermodule commits you make aren't 
reverting or removing those submodules. But you won't fetch the contents 
of those submodules, and won't have them checked out. (Note, however, 
that, if you pull from two places, each of which has changed the same 
submodule you don't have in a different way, you'll get a conflict you 
can't resolve without getting the submodule yourself or getting somebody 
else to merge them; git won't let you make commits which leave the 
submodule merge for somebody who cares)

> If I make a branch on submodule71, the branch is made in all submodules &
> the supermodule.

I don't think this aspect has been worked out too carefully. I think the 
branch is made only in the supermodule, but the submodules are attached to 
the supermodule's index, rather than particularly using branches, and so, 
when the supermodule switches branches, the submodules should all follow, 
regardless of whether they have explicit branches.

> With this setup it should be possible to handle supermodule as a
> normal module with branches for each feature/topic/bugfix, and those
> features being merged into other branches in a reasonable way. Does
> something like this look doable?

AFAIK, supermodules haven't gotten heavy enough use to work out all of the 
details of what should happen when the user does different things. I think 
it's all doable, but you may have to convince people to actually do it, 
and I don't think everything yet works the way you'd want.

	-Daniel
*This .sig left intentionally blank*

Re: Partial checkouts / submodules

[Finn Arne Gangstad]

On Tue, Nov 20, 2007 at 06:33:50PM +0100, Sven Verdoolaege wrote:

> Just "submodule init" and "submodule update" these submodules and
> it looks like you would get what you want...
> 
> > If I make a branch on submodule71, the branch is made in all submodules &
> > the supermodule.
> 
> ... except this one.
> It's not clear why you would even want this.

I'll try to boil this down to the simplest case possible. If
submodules can do this I'll be really happy :)

Developer A makes a change in submodule1 and in submodule2
Developer B makes a change in submodule2 and in submodule3

A and B don't know about eachother. They send their modifications
somewhere (push to a shared repository with a well chosen branch name,
for example), or send a mail "please pull from my repo" to the patch
queue manager.

It is absolutely crucial that for each developer, either both their
modifications go in, or none of them. Git should make picking only
one of their modifications hard.

Also - it would be very good if the history in the master repo would
match the history in all developers' repositories (as the
modifications are merged in by the patch queue manager). I.e. - you
should see a "gif support" feature branch, see the commits on it, and
finally the merge.

- Finn Arne

Re: Partial checkouts / submodules

[Steven Grimm]

On Nov 20, 2007, at 10:03 AM, Daniel Barkalow wrote:
> This has theoretical problems: it's going to be practically  
> impossible, in
> most cases, to write a commit message that describes changes in three
> submodules (which are sometimes used in the context of a different
> supermodule) as well as the supermodule.

I got the impression from his email that there *are* no other  
supermodules. The submodules are submodules purely to reduce the  
amount of data people have to transfer around, not because they're  
logically distinct from the parent.

Something like "Fix bug #10391 by closing the data file before  
launching the GUI" or something like that would be just as valid a  
commit comment in the submodules here as it would be in the  
supermodules.

Sounds like what he wants is actually partial checkout, but since git  
doesn't (yet) support that, he's emulating it as best he can using  
submodules.

> I think it should be possible and sufficient to have "git commit"  
> able to
> recursively start a "git commit" in submodules (where you'd write a
> separate message suitable for exposure to other supermodules), so  
> you'd
> have to write 4 messages but only type one command line.

If the message in the submodule defaulted to the one from the  
supermodule, and there was a way to just accept it without popping up  
an editor N times, this might solve his problem.

-Steve

Re: Partial checkouts / submodules

[Daniel Barkalow]

On Tue, 20 Nov 2007, Steven Grimm wrote:

> On Nov 20, 2007, at 10:03 AM, Daniel Barkalow wrote:
> >This has theoretical problems: it's going to be practically impossible, in
> >most cases, to write a commit message that describes changes in three
> >submodules (which are sometimes used in the context of a different
> >supermodule) as well as the supermodule.
> 
> I got the impression from his email that there *are* no other supermodules.
> The submodules are submodules purely to reduce the amount of data people have
> to transfer around, not because they're logically distinct from the parent.

He said:

"there are multiple superprojects sharing some of the submodules."

Getting the effect of partial checkouts was the first-listed reason, but 
not the only one. The submodules don't make sense except in the context of 
some supermodule, but there are multiple contexts they each appear in.

	-Daniel
*This .sig left intentionally blank*

Re: Partial checkouts / submodules

[Sven Verdoolaege]

On Tue, Nov 20, 2007 at 07:19:33PM +0100, Finn Arne Gangstad wrote:
> On Tue, Nov 20, 2007 at 06:33:50PM +0100, Sven Verdoolaege wrote:
> 
> > Just "submodule init" and "submodule update" these submodules and
> > it looks like you would get what you want...
> > 
> > > If I make a branch on submodule71, the branch is made in all submodules &
> > > the supermodule.
> > 
> > ... except this one.
> > It's not clear why you would even want this.
> 
> I'll try to boil this down to the simplest case possible. If
> submodules can do this I'll be really happy :)
> 
> Developer A makes a change in submodule1 and in submodule2
> Developer B makes a change in submodule2 and in submodule3

I'm assuming that A and B also make a commit to the supermodule
incorporating their changes to the submodules (as in your
original message).

> A and B don't know about eachother. They send their modifications
> somewhere (push to a shared repository with a well chosen branch name,
> for example), or send a mail "please pull from my repo" to the patch
> queue manager.

The commit A made to the superproject contains his changes
to the submodule, so if someone pulls this superproject commit,
she'll get the submodule changes too.
(Although she may choose not to have some or all of these
submodules checked out.)

> Also - it would be very good if the history in the master repo would
> match the history in all developers' repositories (as the
> modifications are merged in by the patch queue manager). I.e. - you
> should see a "gif support" feature branch, see the commits on it, and
> finally the merge.

You can't change the history in git. [*1]

It's still not clear why you would want new branches to be created
in other modules when you create a new branch in some module.

skimo

[*1] There are tools that allow you to create a new history based
on an old history, but it will be a different history.

Re: Partial checkouts / submodules

[Sergei Organov]

Steven Grimm <koreth@midwinter.com> writes:

> On Nov 20, 2007, at 10:03 AM, Daniel Barkalow wrote:
>> This has theoretical problems: it's going to be practically
>> impossible, in
>> most cases, to write a commit message that describes changes in three
>> submodules (which are sometimes used in the context of a different
>> supermodule) as well as the supermodule.
>
> I got the impression from his email that there *are* no other
> supermodules. The submodules are submodules purely to reduce the
> amount of data people have to transfer around, not because they're
> logically distinct from the parent.

I got the same impression, and then I wonder if the next logical thing
the OP will need is, say, support for content moves between
submodules. Somehow I doubt git will ever support that.

-- 
Sergei.

Re: Partial checkouts / submodules

[Daniel Barkalow]

On Tue, 20 Nov 2007, Finn Arne Gangstad wrote:

> On Tue, Nov 20, 2007 at 06:33:50PM +0100, Sven Verdoolaege wrote:
> 
> > Just "submodule init" and "submodule update" these submodules and
> > it looks like you would get what you want...
> > 
> > > If I make a branch on submodule71, the branch is made in all submodules &
> > > the supermodule.
> > 
> > ... except this one.
> > It's not clear why you would even want this.
> 
> I'll try to boil this down to the simplest case possible. If
> submodules can do this I'll be really happy :)
> 
> Developer A makes a change in submodule1 and in submodule2
> Developer B makes a change in submodule2 and in submodule3
> 
> A and B don't know about eachother. They send their modifications
> somewhere (push to a shared repository with a well chosen branch name,
> for example), or send a mail "please pull from my repo" to the patch
> queue manager.
> 
> It is absolutely crucial that for each developer, either both their
> modifications go in, or none of them. Git should make picking only
> one of their modifications hard.

This is the case; if developer A changes 2 from 2-O to 2-A, and developer 
B changes 2 from 2-O to 2-B, merging both supermodule commits gets a 
conflict, which requires a merge in submodule 2 before the supermodule 
merge can be committed.

On the other hand, 3-B will sort of enter the history of submodule 3 
without the submodule 2 merge getting done; however, there's no way for 
anybody to find it if it's not referenced by any supermodule commit yet 
and people don't look at the individual histories of submodules.

One thing that might be an issue is that, if developer A of supermodule I 
makes a commit that changes submodule 1 and submodule 2, and developer B 
on supermodule II decides to merge supermodule I's submodule 1, and 
supermodule II also includes submodule 2, but developer B doesn't care 
about it, supermodule II could end up with only half of developer A's 
change.

> Also - it would be very good if the history in the master repo would
> match the history in all developers' repositories (as the
> modifications are merged in by the patch queue manager). I.e. - you
> should see a "gif support" feature branch, see the commits on it, and
> finally the merge.

This is independant of submodule support, and depends on the patch queue 
manager's policy. In some cases, it's desirable to simplify the history of 
the feature branch when it's being merged into the master repo, so that 
the master repo gets an idealized version of the feature branch (i.e., 
bugs introduced early in the development of the branch and fixed later, 
but never affected the master repo, are not introduced in the first place; 
also, the historical accident of the work on the topic being started 
before other features but completed after them can be smoothed over, with 
the resolution of merge conflicts distributed back to the sites where the 
second set of changes was made). If the patch queue manager does this sort 
of thing, the master repo's history will be different from the feature 
branch's history as it appeared to the developers at the time, but the 
feature branch also generally goes away at this point anyway, so it 
doesn't matter too much.

	-Daniel
*This .sig left intentionally blank*

Re: Partial checkouts / submodules

[Daniel Barkalow]

On Tue, 20 Nov 2007, Sergei Organov wrote:

> I got the same impression, and then I wonder if the next logical thing
> the OP will need is, say, support for content moves between
> submodules. Somehow I doubt git will ever support that.

I don't see why not. If you tell the diff engine to look for content 
moves, and you tell it to descend into submodules that you have, and you 
have both ends of the content move, and you're looking at a supermodule 
commit where such a content move happened, it should show it.

If any of these aren't true (and I don't think we have a "descend into 
available submodules" option currently), you won't see it as a content 
move, but you wouldn't expect to; you don't want to get a diff that says: 
"move from (path you don't have) to (path you have)", with the diff 
showing changes in the process, when you didn't have the original.

	-Daniel
*This .sig left intentionally blank*

Re: Partial checkouts / submodules

[Finn Arne Gangstad]

On Tue, Nov 20, 2007 at 01:59:41PM -0500, Daniel Barkalow wrote:
> On Tue, 20 Nov 2007, Finn Arne Gangstad wrote:
> 
> > I'll try to boil this down to the simplest case possible. If
> > submodules can do this I'll be really happy :)
> > 
> > Developer A makes a change in submodule1 and in submodule2
> > Developer B makes a change in submodule2 and in submodule3
> > 
> > A and B don't know about eachother. They send their modifications
> > somewhere (push to a shared repository with a well chosen branch name,
> > for example), or send a mail "please pull from my repo" to the patch
> > queue manager.
> > 
> > It is absolutely crucial that for each developer, either both their
> > modifications go in, or none of them. Git should make picking only
> > one of their modifications hard.
> 
> This is the case; if developer A changes 2 from 2-O to 2-A, and developer 
> B changes 2 from 2-O to 2-B, merging both supermodule commits gets a 
> conflict, which requires a merge in submodule 2 before the supermodule 
> merge can be committed.

And this is partly why I wanted to branch all the involved modules: In
~99% of the cases, 2-A and 2-B modify different files, or at least
wildly different parts of the same file, so the merge should be
trivial/automatic. Therefore, the supermodule merge should also be a
trivial/automatic merge - but it isn't, is it?

- Finn Arne

Re: Partial checkouts / submodules

[Daniel Barkalow]

On Tue, 20 Nov 2007, Finn Arne Gangstad wrote:

> On Tue, Nov 20, 2007 at 01:59:41PM -0500, Daniel Barkalow wrote:
> > On Tue, 20 Nov 2007, Finn Arne Gangstad wrote:
> > 
> > > I'll try to boil this down to the simplest case possible. If
> > > submodules can do this I'll be really happy :)
> > > 
> > > Developer A makes a change in submodule1 and in submodule2
> > > Developer B makes a change in submodule2 and in submodule3
> > > 
> > > A and B don't know about eachother. They send their modifications
> > > somewhere (push to a shared repository with a well chosen branch name,
> > > for example), or send a mail "please pull from my repo" to the patch
> > > queue manager.
> > > 
> > > It is absolutely crucial that for each developer, either both their
> > > modifications go in, or none of them. Git should make picking only
> > > one of their modifications hard.
> > 
> > This is the case; if developer A changes 2 from 2-O to 2-A, and developer 
> > B changes 2 from 2-O to 2-B, merging both supermodule commits gets a 
> > conflict, which requires a merge in submodule 2 before the supermodule 
> > merge can be committed.
> 
> And this is partly why I wanted to branch all the involved modules: In
> ~99% of the cases, 2-A and 2-B modify different files, or at least
> wildly different parts of the same file, so the merge should be
> trivial/automatic. Therefore, the supermodule merge should also be a
> trivial/automatic merge - but it isn't, is it?

It's an automatic merge if you've got the submodule; if you don't have the 
submodule, you can't tell that the merge would be trivial, because you 
can't even see the order of the history in the submodule, let alone which 
files change and how. This just means that the patch queue manager is 
going to have to have all of the submodules, which is probably reasonable 
if the patch queue manager is responsible for making sure that the 
combination works.

	-Daniel
*This .sig left intentionally blank*

Re: Partial checkouts / submodules

[Jakub Narebski]

Finn Arne Gangstad wrote:

> I'll try to boil this down to the simplest case possible. If
> submodules can do this I'll be really happy :)
> 
> Developer A makes a change in submodule1 and in submodule2
> Developer B makes a change in submodule2 and in submodule3

And committed changes to submodules and supermodule, I guess,
so developer A has submodule1 in state 1a, submodule2 in state 2a.
sumbodule3 in state 3, and supermodule in state [1a, 2a, 3], while
developer B has submodule1 in state 1, submodule2 in state 2b, 
submodule3 in state 3b, and supermodule in state [1, 2b, 3b].
 
> A and B don't know about eachother. They send their modifications
> somewhere (push to a shared repository with a well chosen branch name,
> for example), or send a mail "please pull from my repo" to the patch
> queue manager.
> 
> It is absolutely crucial that for each developer, either both their
> modifications go in, or none of them. Git should make picking only
> one of their modifications hard.

Git treats submodules as whole, so merging A and B supermodules will result
in [1a, CONFLICT(2a/2b), 3b]. 2a/2b _might_ resolve cleanly to 2ab,
but this requires submodule2 to be checked out. I'm not sure what git does
in such case, but it is not much different from the case when both sides A
and B modified the same file, but it merges on file-level cleanly; here we
have subprojects and tree-level in-subproject clean merge.

[...]

By the way, submodules in supermodule should be thought as on 
'detached HEAD'.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git