Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Tim Harper]

OVERVIEW:
On the Git TextMate Bundle, I've automated a lot of the submodule  
commands to make them not a terrible pain to work with. (don't get me  
wrong - I really like the git submodule implementation, I just don't  
like how hard it is to work with).

"WARTS" WITH EXISTING IMPLEMENTATION
1) The submodule stays in the working copy when changing to a branch  
that does not have a submodule.  This can break a build and cause  
problems.  To work around, I have to delete the folder completely (git- 
clean).  Then, when I switch back to the branch again, I have to re- 
download the submodule.
2) I have to type "git checkout branch && git submodule init && git  
submodule update" to be sure that I really have the whole contents of  
the branch.  That's 3 commands, and a lot of typing.
3) If I don't run "git submodule update", and carelessly run "git  
commit -a" or "git add .", I risk propagating a submodule version from  
another branch or undoing an important change.

SUGGESTED ALGORITHM (AS HAS BEEN IMPLEMENTED IN THE GIT TEXTMATE BUNDLE)
When pulling / merging / changing branches:
1) cache all submodules to ~/.git/submodules_cache
    a) move from the working directory to a folder that is a MD5 hex- 
hash of both the submodule path and the submodule url
2) execute the pull / merge / branch change
3) restore all defined submodules to ~/.git/submodules_cache (only the  
submodules that are still defined after the merge / change / pull)
4) execute git submodule init && git submodule update


PITFALLS:
pitfall)
If you commit a change on a submodule that's not on a branch, auto- 
updating submodules will make it difficult to revive that change.

workaround)
Don't allow the user to commit unless they are on a branch.

... couldn't think of anymore.  Anyone?

CONCLUSION
So far, this algorithm holds up well in my use cases, and has made  
submodule management seamless for me (I don't have to know that I'm  
working with submodules).  It's resolved every one of the above  
outlined interface warts.

Would it be a good idea to build this algorithm into git?  What would  
be the best approach?  Am I completely overlooking something by  
designing the Git TextMate bundle to handle submodules this way?

Thanks,
Tim

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Tim Harper]

clarification:

SUGGESTED ALGORITHM (AS HAS BEEN IMPLEMENTED IN THE GIT TEXTMATE BUNDLE)
When pulling / merging / changing branches:
1) cache all submodules to .git/submodules_cache
-  a) move the submodule to .git/submodules_cache/`echo  
"$submodule_path\n$submodule_url" | md5`
2) execute the pull / merge / branch change
3) restore all defined submodules from .git/submodules_cache to their  
respective working directory locations (only the submodules that are  
still defined after the merge / change / pull)
4) execute git "submodule init && git submodule update"


On Apr 29, 2008, at 10:08 PM, Tim Harper wrote:

> OVERVIEW:
> On the Git TextMate Bundle, I've automated a lot of the submodule  
> commands to make them not a terrible pain to work with. (don't get  
> me wrong - I really like the git submodule implementation, I just  
> don't like how hard it is to work with).
>
> "WARTS" WITH EXISTING IMPLEMENTATION
> 1) The submodule stays in the working copy when changing to a branch  
> that does not have a submodule.  This can break a build and cause  
> problems.  To work around, I have to delete the folder completely  
> (git-clean).  Then, when I switch back to the branch again, I have  
> to re-download the submodule.
> 2) I have to type "git checkout branch && git submodule init && git  
> submodule update" to be sure that I really have the whole contents  
> of the branch.  That's 3 commands, and a lot of typing.
> 3) If I don't run "git submodule update", and carelessly run "git  
> commit -a" or "git add .", I risk propagating a submodule version  
> from another branch or undoing an important change.
>
> SUGGESTED ALGORITHM (AS HAS BEEN IMPLEMENTED IN THE GIT TEXTMATE  
> BUNDLE)
> When pulling / merging / changing branches:
> 1) cache all submodules to ~/.git/submodules_cache
>   a) move from the working directory to a folder that is a MD5 hex- 
> hash of both the submodule path and the submodule url
> 2) execute the pull / merge / branch change
> 3) restore all defined submodules to ~/.git/submodules_cache (only  
> the submodules that are still defined after the merge / change / pull)
> 4) execute git submodule init && git submodule update
>
>
> PITFALLS:
> pitfall)
> If you commit a change on a submodule that's not on a branch, auto- 
> updating submodules will make it difficult to revive that change.
>
> workaround)
> Don't allow the user to commit unless they are on a branch.
>
> ... couldn't think of anymore.  Anyone?
>
> CONCLUSION
> So far, this algorithm holds up well in my use cases, and has made  
> submodule management seamless for me (I don't have to know that I'm  
> working with submodules).  It's resolved every one of the above  
> outlined interface warts.
>
> Would it be a good idea to build this algorithm into git?  What  
> would be the best approach?  Am I completely overlooking something  
> by designing the Git TextMate bundle to handle submodules this way?
>
> Thanks,
> Tim

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Andreas Ericsson]

Tim Harper wrote:
> OVERVIEW:
> On the Git TextMate Bundle, I've automated a lot of the submodule 
> commands to make them not a terrible pain to work with. (don't get me 
> wrong - I really like the git submodule implementation, I just don't 
> like how hard it is to work with).
> 
> "WARTS" WITH EXISTING IMPLEMENTATION
> 1) The submodule stays in the working copy when changing to a branch 
> that does not have a submodule.  This can break a build and cause 
> problems.  To work around, I have to delete the folder completely 
> (git-clean).  Then, when I switch back to the branch again, I have to 
> re-download the submodule.
> 2) I have to type "git checkout branch && git submodule init && git 
> submodule update" to be sure that I really have the whole contents of 
> the branch.  That's 3 commands, and a lot of typing.
> 3) If I don't run "git submodule update", and carelessly run "git commit 
> -a" or "git add .", I risk propagating a submodule version from another 
> branch or undoing an important change.
> 
> SUGGESTED ALGORITHM (AS HAS BEEN IMPLEMENTED IN THE GIT TEXTMATE BUNDLE)
> When pulling / merging / changing branches:
> 1) cache all submodules to ~/.git/submodules_cache
>    a) move from the working directory to a folder that is a MD5 hex-hash 
> of both the submodule path and the submodule url
> 2) execute the pull / merge / branch change
> 3) restore all defined submodules to ~/.git/submodules_cache (only the 
> submodules that are still defined after the merge / change / pull)
> 4) execute git submodule init && git submodule update
> 
> 
> PITFALLS:
> pitfall)
> If you commit a change on a submodule that's not on a branch, 
> auto-updating submodules will make it difficult to revive that change.
> 
> workaround)
> Don't allow the user to commit unless they are on a branch.
> 
> ... couldn't think of anymore.  Anyone?
> 
> CONCLUSION
> So far, this algorithm holds up well in my use cases, and has made 
> submodule management seamless for me (I don't have to know that I'm 
> working with submodules).  It's resolved every one of the above outlined 
> interface warts.
> 
> Would it be a good idea to build this algorithm into git?  What would be 
> the best approach?  Am I completely overlooking something by designing 
> the Git TextMate bundle to handle submodules this way?
> 

I don't use submodules at the moment, but I have several "lib-ish" pieces
of code that would benefit greatly from becoming submodules. The not-exactly
seamlessness of them has so far been a hindrance though, but it sounds as
if your changes (assuming they don't affect anything else) should make lessen
the submodule headaches somewhat.

So, where be the patches?

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Johannes Schindelin]

Hi,

On Tue, 29 Apr 2008, Tim Harper wrote:

> 1) The submodule stays in the working copy when changing to a branch 
>    that does not have a submodule.  This can break a build and cause 
>    problems.  To work around, I have to delete the folder completely 
>    (git-clean).  Then, when I switch back to the branch again, I have to 
>    re-download the submodule.

The problem, of course, is that you can easily have valuable, but 
not-tracked, files in there.  Deleting the submodule is therefore no 
option.

> 2) I have to type "git checkout branch && git submodule init && git 
>    submodule update" to be sure that I really have the whole contents of 
>    the branch.  That's 3 commands, and a lot of typing.

There is no way around "checkout branch", and I think that is a good 
thing.

But once you did "submodule init", you will never need to run it again, 
since it edits your .git/config, which does not change when switching 
branches.

And as for "submodule update", I like the fact that the submodule is not 
updated automatically.  For example, when I actively develop a submodule, 
but have to rebase the superproject, I would _hate_ it if the submodule 
wass updated.

The whole idea about submodules is that they are repositories of their own 
right, and therefore the superproject should not mess with them, _unless_ 
explicitely asked to, with "submodule update".

> 3) If I don't run "git submodule update", and carelessly run "git commit 
>    -a"  or "git add .", I risk propagating a submodule version from 
>    another branch or undoing an important change.

git commit -a is something that might make sense for newbies, but you 
really should learn to use git add -p and commit without -a.

> SUGGESTED ALGORITHM (AS HAS BEEN IMPLEMENTED IN THE GIT TEXTMATE BUNDLE)
> When pulling / merging / changing branches:
> 1) cache all submodules to ~/.git/submodules_cache
>   a) move from the working directory to a folder that is a MD5 hex-hash of
> both the submodule path and the submodule url
> 2) execute the pull / merge / branch change
> 3) restore all defined submodules to ~/.git/submodules_cache (only the
> submodules that are still defined after the merge / change / pull)
> 4) execute git submodule init && git submodule update
> 
> 
> PITFALLS:
> pitfall)
> If you commit a change on a submodule that's not on a branch, auto-updating
> submodules will make it difficult to revive that change.
> 
> workaround)
> Don't allow the user to commit unless they are on a branch.
> 
> ... couldn't think of anymore.  Anyone?

I do not like that.  I think that the user should be responsible to take 
care of the up-to-dateness of the submodule.  As far as the superproject 
is concerned, it just keeps track of the committed submodule state, but 
does not enforce it.

FWIW I restarted on my --ignore-submodules patch, which can be seen here:

http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/my-next

It lacks tests, therefore it has not been submitted for review yet.

The basic idea is that not only "checkout" ignores the _contents_ of the 
submodules, but optionally "diff", and using that option "stash" and 
"rebase".

IOW you can "stash apply" even if the submodule is dirty.  Which makes 
sense, given that "stash" succeeds, too...

Ciao,
Dscho

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Avery Pennarun]

On 4/30/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  On Tue, 29 Apr 2008, Tim Harper wrote:
>
>  > 1) The submodule stays in the working copy when changing to a branch
>  >    that does not have a submodule.  This can break a build and cause
>  >    problems.  To work around, I have to delete the folder completely
>  >    (git-clean).  Then, when I switch back to the branch again, I have to
>  >    re-download the submodule.
>
> The problem, of course, is that you can easily have valuable, but
>  not-tracked, files in there.  Deleting the submodule is therefore no
>  option.

I agree that blindly deleting isn't a good choice.  For example, what
if I've checked stuff into the submodule but never pushed it anywhere?
 Ugh.

That said, it makes submodule folders act completely inconsistently
with normal folders, which is highly undesirable.  The current
behaviour strongly encourages me to avoid submodules when I would
otherwise like to use them, just to keep the rest of my team members
(who are not git experts) from going insane.

>  But once you did "submodule init", you will never need to run it again,
>  since it edits your .git/config, which does not change when switching
>  branches.

Not true.  If ".gitmodules" is different between branches, then
.git/config will have the wrong information.  I think this was the
reason for the "read .gitmodules directly and don't worry about
.git/config" discussion/patches earlier.

>  And as for "submodule update", I like the fact that the submodule is not
>  updated automatically.  For example, when I actively develop a submodule,
>  but have to rebase the superproject, I would _hate_ it if the submodule
>  wass updated.

Why?  Every other folder in your entire project gets updated when you
"git checkout".  Why are submodules different?  I can personally vouch
that this is confusing for almost everyone I've seen who tried it.

>  The whole idea about submodules is that they are repositories of their own
>  right, and therefore the superproject should not mess with them, _unless_
>  explicitely asked to, with "submodule update".

I think perhaps this is the root of the problem; you're thinking of
the superproject as (A) "a tool for helping me track multiple
subprojects automatically", while I think most people (or at least I)
think of submodules as (B) "folders that have their own series of
commits and thus can be shared between projects."

If you think of it as (A), it's inconvenient to have them shuffling
around automatically.  But if you're in the (B) camp, having it not
update automatically seems insane.

I actually don't know any use cases for (A).  If my app depends on a
library in a submodule, for example, and I switch to a branch of my
app that uses a different version of the submodule, why would I ever
*not* want the submodule to switch too?  If it doesn't, it'll probably
break the compile.

>  > 3) If I don't run "git submodule update", and carelessly run "git commit
>  >    -a"  or "git add .", I risk propagating a submodule version from
>  >    another branch or undoing an important change.
>
> git commit -a is something that might make sense for newbies, but you
>  really should learn to use git add -p and commit without -a.

This is a matter of opinion; pretty much every other major VCS does
the equivalent of "git commit -a" when you commit, and most people
don't seem to mind.  It is therefore possible that git is the weird
one, not every other VCS.

In particular, in this case what git is doing is, "You checked out a
new branch, but I don't really trust you, so I actually made this
modification in your tree for you.  Now if you commit, you can check
in that change that I think you meant to check in."  Wouldn't you
expect "git checkout branch; git commit -a" on a clean tree to never
commit anything?

>  > PITFALLS:
>  > pitfall)
>  > If you commit a change on a submodule that's not on a branch, auto-updating
>  > submodules will make it difficult to revive that change.
>  >
>  > workaround)
>  > Don't allow the user to commit unless they are on a branch.
>  >
>  > ... couldn't think of anymore.  Anyone?

We had some discussion on the list earlier about having submodule
checkouts automatically acquire a branch name, so that commits don't
get lost as easily.  I was going to think about this more and
eventually submit a patch, but I haven't gotten to it yet.  Anyway,
the idea is that you have a branch by default, so that you don't end
up in the useless situation of not being on a branch, which encourages
checking in without being on a branch, in the first place.

Have fun,

Avery

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Ping Yin]

On Thu, May 1, 2008 at 12:47 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On 4/30/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>  >  On Tue, 29 Apr 2008, Tim Harper wrote:

>
>  Not true.  If ".gitmodules" is different between branches, then
>  .git/config will have the wrong information.  I think this was the
>  reason for the "read .gitmodules directly and don't worry about
>  .git/config" discussion/patches earlier.

I tried hard to avoid 'git submodule init' in that series.
Unfortuately, few people agreed with me.


>  We had some discussion on the list earlier about having submodule
>  checkouts automatically acquire a branch name, so that commits don't
>  get lost as easily.  I was going to think about this more and
>  eventually submit a patch, but I haven't gotten to it yet.  Anyway,
>  the idea is that you have a branch by default, so that you don't end
>  up in the useless situation of not being on a branch, which encourages
>  checking in without being on a branch, in the first place.
>

I like this idea and it is very useful for me.


-- 
Ping Yin

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Roman Shaposhnik]

On Wed, 2008-04-30 at 12:47 -0400, Avery Pennarun wrote:
> On 4/30/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> That said, it makes submodule folders act completely inconsistently
> with normal folders, which is highly undesirable.  The current
> behaviour strongly encourages me to avoid submodules when I would
> otherwise like to use them, just to keep the rest of my team members
> (who are not git experts) from going insane.

Well said. I find myself in exactly the same situation wrt. submodules.
To give an analogy imagine that everytime you had a mount-point in
your path you'd have to do something special in order to cross it.
IOW, open("/tmp/foo/bar/baz.c") would just work, where
open("/mnt/point1/foo.c") would require you to first cross /mnt/point1
somehow. Currently submodules feel very much like the second case
to me. Now, I've shut up for the time being on that other thread,
since I'd like to come up with a clearly defined workflow first.
But for the record: the way submodules are implemented right now
in a porcelain make them pretty much an impossible sell to those
in my org who don't want to know much about SCMs.

> >  But once you did "submodule init", you will never need to run it again,
> >  since it edits your .git/config, which does not change when switching
> >  branches.
> 
> Not true.  If ".gitmodules" is different between branches, then
> .git/config will have the wrong information.  I think this was the
> reason for the "read .gitmodules directly and don't worry about
> .git/config" discussion/patches earlier.

Exactly!

> >  And as for "submodule update", I like the fact that the submodule is not
> >  updated automatically.  For example, when I actively develop a submodule,
> >  but have to rebase the superproject, I would _hate_ it if the submodule
> >  wass updated.
> 
> Why?  Every other folder in your entire project gets updated when you
> "git checkout".  Why are submodules different?  I can personally vouch
> that this is confusing for almost everyone I've seen who tried it.

+1 ;-)


> >  > PITFALLS:
> >  > pitfall)
> >  > If you commit a change on a submodule that's not on a branch, auto-updating
> >  > submodules will make it difficult to revive that change.
> >  >
> >  > workaround)
> >  > Don't allow the user to commit unless they are on a branch.
> >  >
> >  > ... couldn't think of anymore.  Anyone?
> 
> We had some discussion on the list earlier about having submodule
> checkouts automatically acquire a branch name, so that commits don't
> get lost as easily.  I was going to think about this more and
> eventually submit a patch, but I haven't gotten to it yet.  Anyway,
> the idea is that you have a branch by default, so that you don't end
> up in the useless situation of not being on a branch, which encourages
> checking in without being on a branch, in the first place.

Can you give a gmane pointer to that thread?

Thanks,
Roman.

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Avery Pennarun]

On 4/30/08, Roman Shaposhnik <rvs@sun.com> wrote:
> Well said.

Thanks.  I'm having a bad day so a little agreement helps a lot :)

>  > We had some discussion on the list earlier about having submodule
>  > checkouts automatically acquire a branch name, so that commits don't
>  > get lost as easily.  I was going to think about this more and
>  > eventually submit a patch, but I haven't gotten to it yet.  Anyway,
>  > the idea is that you have a branch by default, so that you don't end
>  > up in the useless situation of not being on a branch, which encourages
>  > checking in without being on a branch, in the first place.
>
> Can you give a gmane pointer to that thread?

Try this:
http://article.gmane.org/gmane.comp.version-control.git/78675

Note that this particular message is representative, but is not the
best of the solutions proposed during that thread.  Exactly which of
the solutions, if any, is the best is left as an exercise to the
reader :)

Have fun,

Avery

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Tim Harper]

I had a feeling my proposal would be met with a mix of opposition and  
acceptance, depending on which "git camp" you are currently in.

On Apr 30, 2008, at 4:31 AM, Johannes Schindelin wrote:
> Hi,
>
> On Tue, 29 Apr 2008, Tim Harper wrote:
>
>> 1) The submodule stays in the working copy when changing to a branch
>>   that does not have a submodule.  This can break a build and cause
>>   problems.  To work around, I have to delete the folder completely
>>   (git-clean).  Then, when I switch back to the branch again, I  
>> have to
>>   re-download the submodule.
>
> The problem, of course, is that you can easily have valuable, but
> not-tracked, files in there.  Deleting the submodule is therefore no
> option.
>

Submodules are not deleted.  They are moved out of the working copy  
into a folder in .git.  Therefore, upon changing back to the branch  
with the submodule, they are restored, without nay a hair on their  
head lost.

>> 2) I have to type "git checkout branch && git submodule init && git
>>   submodule update" to be sure that I really have the whole  
>> contents of
>>   the branch.  That's 3 commands, and a lot of typing.
>
> There is no way around "checkout branch", and I think that is a good
> thing.
>

Naturally


> But once you did "submodule init", you will never need to run it  
> again,
> since it edits your .git/config, which does not change when switching
> branches.
>

If working in a collaborative environment, when someone adds a new  
submodule, it's extra burden to have to know "oh, there's a new  
submodule now, so I have to run "git submodule init".  I find it's  
more than sufficient just to run it everytime, and costs me nothing.   
Why aren't "gut submodule update" and "git submodule init" the same  
command?  Perhaps a compromise could be met with "git submodule update  
-i".


> And as for "submodule update", I like the fact that the submodule is  
> not
> updated automatically.  For example, when I actively develop a  
> submodule,
> but have to rebase the superproject, I would _hate_ it if the  
> submodule
> wass updated.
>

Then we could make my proposed idea a configuration variable :)  For  
my use case, I passionately dislike the fact that a submodule is not  
updated automatically.  There's never a time when I don't want to  
update the submodule.  The submodule is a very important piece of our  
project and the super-project depends on it being at the right version.

I suspect a large majority of git users who would use submodules, and  
a handful of those already using it, have a similar use case.  Though,  
I have no evidence to support this, other than my own observations.

> The whole idea about submodules is that they are repositories of  
> their own
> right, and therefore the superproject should not mess with them,  
> _unless_
> explicitely asked to, with "submodule update".
>
>> 3) If I don't run "git submodule update", and carelessly run "git  
>> commit
>>   -a"  or "git add .", I risk propagating a submodule version from
>>   another branch or undoing an important change.
>
> git commit -a is something that might make sense for newbies, but you
> really should learn to use git add -p and commit without -a.
>

Or... use my textmate bundle for git, which makes committing very  
easy. :)  The newbies are pointed to use "git add ." - just try typing  
"git add"

>> SUGGESTED ALGORITHM (AS HAS BEEN IMPLEMENTED IN THE GIT TEXTMATE  
>> BUNDLE)
>> When pulling / merging / changing branches:
>> 1) cache all submodules to ~/.git/submodules_cache
>> a) move from the working directory to a folder that is a MD5 hex- 
>> hash of
>> both the submodule path and the submodule url
>> 2) execute the pull / merge / branch change
>> 3) restore all defined submodules to ~/.git/submodules_cache (only  
>> the
>> submodules that are still defined after the merge / change / pull)
>> 4) execute git submodule init && git submodule update
>>
>>
>> PITFALLS:
>> pitfall)
>> If you commit a change on a submodule that's not on a branch, auto- 
>> updating
>> submodules will make it difficult to revive that change.
>>
>> workaround)
>> Don't allow the user to commit unless they are on a branch.
>>
>> ... couldn't think of anymore.  Anyone?
>
> I do not like that.  I think that the user should be responsible to  
> take
> care of the up-to-dateness of the submodule.  As far as the  
> superproject
> is concerned, it just keeps track of the committed submodule state,  
> but
> does not enforce it.
>

Does git serve me or do I serve git?  The whole point of software is  
to make our lives easier and more productive.  If submodules are apart  
of our project, why would not want git to handle making sure your  
submodules are at the right version?

I could see an issue if you changed the submodule version, and forgot  
to commit the version change in your super project, and then switched  
branches, and lost your version change.  Perhaps the correct behavior  
in this case would be to only automatically update the submodule  
version if it didn't differ from the superproject version in the  
"from" branch.

> FWIW I restarted on my --ignore-submodules patch, which can be seen  
> here:
>
> http://repo.or.cz/w/git/dscho.git?a=shortlog;h=refs/heads/my-next
>
> It lacks tests, therefore it has not been submitted for review yet.
>
> The basic idea is that not only "checkout" ignores the _contents_ of  
> the
> submodules, but optionally "diff", and using that option "stash" and
> "rebase".
>
> IOW you can "stash a

Leaving me hanging?  lol.

Thanks,
Tim

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Avery Pennarun]

On 4/30/08, Tim Harper <timcharper@gmail.com> wrote:
> > The problem, of course, is that you can easily have valuable, but
> > not-tracked, files in there.  Deleting the submodule is therefore no
> > option.
>
>  Submodules are not deleted.  They are moved out of the working copy into a
> folder in .git.  Therefore, upon changing back to the branch with the
> submodule, they are restored, without nay a hair on their head lost.

<drool>

Yes!  I'm a little sad that I didn't think of that, because it sounds
like *exactly* what I want.

What about the following case:
- submodule matches the super module checkin
- make changes to submodule but to *not* commit them
- switch supermodule branches, which should checkout a different submodule
- submodule checkout causes a conflict with uncommitted files

What will/should happen here?  It seems like either the supermodule's
submodule pointer won't be set properly (ie. git-submodule-update will
fail, but the supermodule won't be marked as conflicted, thus
git-commit in the supermodule will commit the wrong submodule
revision) or else submodule files might have to be lost or something.

Also, someone earlier asked for a link to your work.  I'd like to see
it too, as I don't know what a "textmate git bundle" is.  I gather
textmate is a MacOS X program, but I don't know what that has to do
with git-submodule :)

Thanks,

Avery

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Tim Harper]

On Apr 30, 2008, at 2:31 PM, Avery Pennarun wrote:

> On 4/30/08, Tim Harper <timcharper@gmail.com> wrote:
>>> The problem, of course, is that you can easily have valuable, but
>>> not-tracked, files in there.  Deleting the submodule is therefore no
>>> option.
>>
>> Submodules are not deleted.  They are moved out of the working copy  
>> into a
>> folder in .git.  Therefore, upon changing back to the branch with the
>> submodule, they are restored, without nay a hair on their head lost.
>
> <drool>
>
> Yes!  I'm a little sad that I didn't think of that, because it sounds
> like *exactly* what I want.
>
Glad to hear I'm not the only one who feels that way :)


> What about the following case:
> - submodule matches the super module checkin

?

>
> - make changes to submodule but to *not* commit them

Changes will never be lost, because the submodule is either stashed  
away, working copy and all, or a "checkout" command is issued to  
change the HEAD pointer.  The latter is like changing branches with  
uncommitted code - the changes will either be carried, and possibly  
conflict.
>
> - switch supermodule branches, which should checkout a different  
> submodule

Submodule is automatically changed for you, so changing branches makes  
sure  you always have the "whole project".  Recursion isn't handled at  
the time being, so it only works for 1 level deep.

>
> - submodule checkout causes a conflict with uncommitted files

Oooh - this could be a problem currently, especially if you're not on  
a branch.  If you have to resolve conflicts the way to resolve them is  
to commit, and you can't switch to a branch to commit unless you  
resolve your issues.  In which case, you'll probably want to resolve,  
commit, then create a new branch with your current HEAD, then merge it  
into a branch.  I'll visit that as issues arise.  This is bleeding  
edge experimentation here :)
>
>
> What will/should happen here?  It seems like either the supermodule's
> submodule pointer won't be set properly (ie. git-submodule-update will
> fail, but the supermodule won't be marked as conflicted, thus
> git-commit in the supermodule will commit the wrong submodule
> revision) or else submodule files might have to be lost or something.

This is a good point: I don't believe that submodules have anyway to  
mark the submodule revision as "conflicted".  In order for this  
concept to be handled in its full glory, that will most certainly need  
to be handled.

>
>
> Also, someone earlier asked for a link to your work.  I'd like to see
> it too, as I don't know what a "textmate git bundle" is.  I gather
> textmate is a MacOS X program, but I don't know what that has to do
> with git-submodule :)
>

TextMate is an editor for MacOSX.

Here's a link to the bundle.  It's written in ruby.:
http://github.com/timcharper/git-tmbundle/

What does it have to do with submodules?  It's essentially a "GUI" for  
git.  It provides automation for a lot of common tasks also.  My team  
has a need for submodules, but unfortunately in their current state I  
couldn't recommend it to them, so I "smoothed over" the rough edges by  
automating a lot of the awkwardness.  So far, it's been working well  
for us.

> Thanks,
>
> Avery

Thank you!

Tim

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Avery Pennarun]

On 4/30/08, Tim Harper <timcharper@gmail.com> wrote:
> > What about the following case:
> > - submodule matches the super module checkin
> > - make changes to submodule but to *not* commit them
> > - switch supermodule branches, which should checkout a different submodule
> > - submodule checkout causes a conflict with uncommitted files
> > [all of the above were meant to be steps in the same test case]
>
> Oooh - this could be a problem currently, especially if you're not on a
> branch.  If you have to resolve conflicts the way to resolve them is to
> commit, and you can't switch to a branch to commit unless you resolve your
> issues.  In which case, you'll probably want to resolve, commit, then create
> a new branch with your current HEAD, then merge it into a branch.  I'll
> visit that as issues arise.  This is bleeding edge experimentation here :)

I know how to deal with it manually, but I'm more concerned about what
would happen in the automatic case, that is, the submodule is dirty
and the supermodule tries to switch branches.

I guess the standard thing to do would be to just have git-checkout in
the supermodule check all the submodules first, and if any of them are
dirty, refuse to do anything at all.

>  Here's a link to the bundle.  It's written in ruby.:
>  http://github.com/timcharper/git-tmbundle/
>
> What does it have to do with submodules?  It's essentially a "GUI" for git.
> It provides automation for a lot of common tasks also.  My team has a need
> for submodules, but unfortunately in their current state I couldn't
> recommend it to them, so I "smoothed over" the rough edges by automating a
> lot of the awkwardness.  So far, it's been working well for us.

Hmm, so the bad news is that it doesn't really help me then :(

It would be awesome if you could turn the fancy behaviour of this
bundle into patches to git-submodule, for example, and then have your
textmate macros call the modified git-submodule.  It might be a bit of
an uphill battle to get the patches accepted into the release, but I
think it's worth the effort, as git-submodule in its current state is
just a non-starter for my group at least.

Have fun,

Avery

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Roman Shaposhnik]

On Wed, 2008-04-30 at 17:48 -0400, Avery Pennarun wrote:
> It would be awesome if you could turn the fancy behaviour of this
> bundle into patches to git-submodule, for example, and then have your
> textmate macros call the modified git-submodule.  It might be a bit of
> an uphill battle to get the patches accepted into the release, but I
> think it's worth the effort, as git-submodule in its current state is
> just a non-starter for my group at least.

Doesn't the fact that the workflows around submodules tend to differ so
much call for different incarnations of git-submodule? IOW, wouldn't
it be ok to have an alternative to git-submodule somewhere in contrib?

Thanks,
Roman.

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Avery Pennarun]

On 4/30/08, Roman Shaposhnik <rvs@sun.com> wrote:
> On Wed, 2008-04-30 at 17:48 -0400, Avery Pennarun wrote:
>  > It would be awesome if you could turn the fancy behaviour of this
>  > bundle into patches to git-submodule, for example, and then have your
>  > textmate macros call the modified git-submodule.  It might be a bit of
>  > an uphill battle to get the patches accepted into the release, but I
>  > think it's worth the effort, as git-submodule in its current state is
>  > just a non-starter for my group at least.
>
> Doesn't the fact that the workflows around submodules tend to differ so
>  much call for different incarnations of git-submodule? IOW, wouldn't
>  it be ok to have an alternative to git-submodule somewhere in contrib?

This would be okay with me.  git-submodule itself doesn't seem to do
anything very complex.  I would be happy to help contribute to such a
thing, as my group needs it rather desperately.

Avery

Re: Making submodules easier to work with

[Finn Arne Gangstad]

On Wed, Apr 30, 2008 at 06:28:00PM -0400, Avery Pennarun wrote:
> On 4/30/08, Roman Shaposhnik <rvs@sun.com> wrote:
> > On Wed, 2008-04-30 at 17:48 -0400, Avery Pennarun wrote:
> >  > It would be awesome if you could turn the fancy behaviour of this
> >  > bundle into patches to git-submodule, for example, and then have your
> >  > textmate macros call the modified git-submodule.  It might be a bit of
> >  > an uphill battle to get the patches accepted into the release, but I
> >  > think it's worth the effort, as git-submodule in its current state is
> >  > just a non-starter for my group at least.
> >
> > Doesn't the fact that the workflows around submodules tend to differ so
> >  much call for different incarnations of git-submodule? IOW, wouldn't
> >  it be ok to have an alternative to git-submodule somewhere in contrib?
> 
> This would be okay with me.  git-submodule itself doesn't seem to do
> anything very complex.  I would be happy to help contribute to such a
> thing, as my group needs it rather desperately.

Where do we want to go with submodules?

Today, submodules seem to be a "read-only" implementation of the
supermodule. By that I mean that it is (only?) suited for creating a
supermodule that consists of independently released submodules, where
all development happens in the submodules, and you sometimes update
the supermodule to refer to a new version of a submodule.

What I've tried to achieve with submodules is a bit different: I want
most development to happen in the supermodule _as if_ the submodules
were part of the supermodule. There are two reasons for not doing it
with one big module: Total size can be a bit too big, but most
importantly, some submodules are shared between different super
modules and there is a certain level of synchronizing. Does this match
your scenarios in any way?

Simplified but realistic scenario [*]:

Supermodule "crawler" has some native code, and uses submodule "os-lib".
Supermodule "indexer" has some native code, and uses submodule "os-lib".

One group works on "crawler" and releases a new crawler every 3 months.
Another group works on "indexer" and releases a new indexer every 2 weeks.

The guy who wrote "os-lib" quit 2 years ago, and no one really
maintains it. Once a year or so someone might try to unify the
"os-lib" releases (so it should be POSSIBLE, just not the norm).


What I would like: People should be able to work on "crawler" as if it
was a single git repository, in particular this means:

o Branching "crawler" means branching "os-lib"
o You can send a patch that contains changes both to "crawler" and "os-lib"
  and get it applied in a resonable way as ONE modification (and git-am
  would do the right thing)
o Merging branch a and branch b in "crawler" also merges the matching
  branches a and b in "os-lib".
o Pushing the supermodule also pushes the submodules

The branch names in "os-lib" can (and probably should) be mangled, so
they are a combination of the supermodule and the branch
name. E.g. something like crawler::branch, crawler/branch, or even
{crawler}.branch. (Is there a character that is currently not allowed in
branch names that can be used as an escape?)

Some lose implementation ideas: (code would be better, I know):

- Enable new behaviour with "git subdirectory" instead of "git submodule",
  and let "git submodule" keep the old behaviour.
- subdirectories get the following magic additions:
  - branching in parent creates {parent}.<branch> in subdirectories
  - merging branches in parent merges the corresponding {parent}.branches
    in subdirectories
  - pushing a parent also pushes the subdirectories
  - changing the parent to something that does not contain the subdirectory
    is only allowed if the subdirectory is "clean" (no uncommitted changes).


[*] In reality: there are 300+ submodules, in all states from
"actively maintained" to "last person who understood the code left 10
years ago". There are maybe 5 or 10 supermodules.

- Finn Arne

Re: Making submodules easier to work with

[Avery Pennarun]

On Thu, May 1, 2008 at 2:38 PM, Finn Arne Gangstad <finnag@pvv.org> wrote:
>  Today, submodules seem to be a "read-only" implementation of the
>  supermodule. By that I mean that it is (only?) suited for creating a
>  supermodule that consists of independently released submodules, where
>  all development happens in the submodules, and you sometimes update
>  the supermodule to refer to a new version of a submodule.
>
>  What I've tried to achieve with submodules is a bit different: I want
>  most development to happen in the supermodule _as if_ the submodules
>  were part of the supermodule. There are two reasons for not doing it
>  with one big module: Total size can be a bit too big, but most
>  importantly, some submodules are shared between different super
>  modules and there is a certain level of synchronizing. Does this match
>  your scenarios in any way?

Your version (the second paragraph) matches my usage exactly.  The
first paragraph does not, but I gather from some discussion on this
list that it does match some people's use cases, so I guess it should
continue to be available.

>  o Branching "crawler" means branching "os-lib"
>  o You can send a patch that contains changes both to "crawler" and "os-lib"
>   and get it applied in a resonable way as ONE modification (and git-am
>   would do the right thing)
>  o Merging branch a and branch b in "crawler" also merges the matching
>   branches a and b in "os-lib".
>  o Pushing the supermodule also pushes the submodules

The above would fit fine into my workflow, although it might be more
fancy than I really need.  Personally, I don't mind thinking of my
submodules as separate projects (ie. I should expect to commit,
branch, merge, and push separately).  But if the above features
existed I would adjust my working style to use them, just for the
added day-to-day convenience factor.

Doing things like a single patch against one repo is a bit messy,
because (presumably) you'd have the same commit message in both repos,
which wouldn't really make sense.

>  - Enable new behaviour with "git subdirectory" instead of "git submodule",
>   and let "git submodule" keep the old behaviour.

If we get to the point where patchsets are gettingd sent around to
play with this, is it better to modify git-submodule or to create an
entirely new file?  I don't know the preferred way of doing this.

Have fun,

Avery

Re: Making submodules easier to work with

[Roman Shaposhnik]

On Thu, 2008-05-01 at 15:55 -0400, Avery Pennarun wrote:
> >  o Branching "crawler" means branching "os-lib"
> >  o You can send a patch that contains changes both to "crawler" and "os-lib"
> >   and get it applied in a resonable way as ONE modification (and git-am
> >   would do the right thing)
> >  o Merging branch a and branch b in "crawler" also merges the matching
> >   branches a and b in "os-lib".
> >  o Pushing the supermodule also pushes the submodules
> 
> The above would fit fine into my workflow, although it might be more
> fancy than I really need.  Personally, I don't mind thinking of my
> submodules as separate projects (ie. I should expect to commit,
> branch, merge, and push separately).  But if the above features
> existed I would adjust my working style to use them, just for the
> added day-to-day convenience factor.
> 
> Doing things like a single patch against one repo is a bit messy,
> because (presumably) you'd have the same commit message in both repos,
> which wouldn't really make sense.

May be my brain is saturated with "partial cloning" but somehow the 
following looks like an interesting twist on a decentralized SCM:
imagine that the picture given by Finn Arne Gangstad weren't 
static. IOW, os-lib wasn't really a separate component to begin
with but was first developed as part of a "crawler" and only
when the other team started to implement "indexer" there was
a need for os-lib to be shared between two independent projects.
Is there any nice way to express such a dynamic history sharing,
short of truly refactoring os-lib into a separate Git repository
and treating it either as a submodule or a subtree-merge?

Thanks,
Roman.

Re: Making submodules easier to work with

[Avery Pennarun]

On 5/6/08, Roman Shaposhnik <rvs@sun.com> wrote:
> May be my brain is saturated with "partial cloning" but somehow the
>  following looks like an interesting twist on a decentralized SCM:
>  imagine that the picture given by Finn Arne Gangstad weren't
>  static. IOW, os-lib wasn't really a separate component to begin
>  with but was first developed as part of a "crawler" and only
>  when the other team started to implement "indexer" there was
>  a need for os-lib to be shared between two independent projects.
>  Is there any nice way to express such a dynamic history sharing,
>  short of truly refactoring os-lib into a separate Git repository
>  and treating it either as a submodule or a subtree-merge?

Personally, I think it would be good enough to split out the os-lib
into its own repo using "git-filter-branch --subdirectory-filter", and
then link to it in newer versions of your crawler and indexer projects
using git-submodule.

There would be a bit of wastage here, since crawler still contains the
history of os-lib, which is the same (even the same tree and file
objects!) as the ones in os-lib.

I can think of two responses to that:

1) It's not very important, disk space is cheap and git repositories
are small, and it's much better to have an accurate history of the
crawler project than to overoptimize for space.

2) If the supermodule and submodule shared the same object repository
(eg. the submodule was checked out with
--alternate=<supermodule-gitdir>), there would be no need to waste
storage space.  If/when I get my act in gear and start submitting
git-submodule patches, supporting this behaviour is the direction I'll
probably start in.
(http://article.gmane.org/gmane.comp.version-control.git/78675)

Have fun,

Avery

Re: Making submodules easier to work with

[Ping Yin]

On Thu, May 8, 2008 at 12:14 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On 5/6/08, Roman Shaposhnik <rvs@sun.com> wrote:
>  > May be my brain is saturated with "partial cloning" but somehow the
>  >  following looks like an interesting twist on a decentralized SCM:
>  >  imagine that the picture given by Finn Arne Gangstad weren't
>  >  static. IOW, os-lib wasn't really a separate component to begin
>  >  with but was first developed as part of a "crawler" and only
>  >  when the other team started to implement "indexer" there was
>  >  a need for os-lib to be shared between two independent projects.
>  >  Is there any nice way to express such a dynamic history sharing,
>  >  short of truly refactoring os-lib into a separate Git repository
>  >  and treating it either as a submodule or a subtree-merge?
>
>  Personally, I think it would be good enough to split out the os-lib
>  into its own repo using "git-filter-branch --subdirectory-filter", and
>  then link to it in newer versions of your crawler and indexer projects
>  using git-submodule.
>
>  There would be a bit of wastage here, since crawler still contains the
>  history of os-lib, which is the same (even the same tree and file
>  objects!) as the ones in os-lib.
>
>  I can think of two responses to that:
>
>  1) It's not very important, disk space is cheap and git repositories
>  are small, and it's much better to have an accurate history of the
>  crawler project than to overoptimize for space.
>
>  2) If the supermodule and submodule shared the same object repository
>  (eg. the submodule was checked out with
>  --alternate=<supermodule-gitdir>), there would be no need to waste
>  storage space.  If/when I get my act in gear and start submitting
>  git-submodule patches, supporting this behaviour is the direction I'll
>  probably start in.
>  (http://article.gmane.org/gmane.comp.version-control.git/78675)
>

It doesn't change the workflow whether using --alternative (although i
think it is a good idea).

The most interesting idea in that thread is the "When checking out a
submodule, give the submodule's current commit a useful branch name".
As a heavy submodule user, this will make my life much easier.


-- 
Ping Yin

Re: Making submodules easier to work with

[Steven Grimm]

On May 1, 2008, at 11:38 AM, Finn Arne Gangstad wrote:
> Where do we want to go with submodules?

I have two uses for submodules.

The first one is... well, not really submodules at all. I really want  
sparse checkout so I can avoid bothering with parts of the larger  
project tree I don't care about or need. But right now submodules are  
the only way to approximate sparse checkout, so until we have true  
sparse checkout support, I want to be able to get semantics that act  
like the submodules are just subdirectories in the supermodule with no  
special care required. For this use case, the "automatically update  
all my submodules and push all of them when I push the supermodule"  
behavior being discussed here, along with synchronizing local branches  
between the super and submodules, would be a huge usability improvement.

The second is closer to the current design's intended use case: a sort  
of version-controlled symbolic link. This is for libraries and other  
external dependencies. The current model is not too bad for this use  
case, though ideally I'd like to see it get as low-touch as, say, svn  
externals, where only one person on a development team needs to worry  
about setting up and updating submodules and they cause the right  
files to show up automatically for everyone else.

-Steve

Re: Making submodules easier to work with

[Roman Shaposhnik]

On Thu, 2008-05-01 at 16:29 -0700, Steven Grimm wrote:
> On May 1, 2008, at 11:38 AM, Finn Arne Gangstad wrote:
> > Where do we want to go with submodules?
> 
> I have two uses for submodules.
> 
> The first one is... well, not really submodules at all. I really want  
> sparse checkout so I can avoid bothering with parts of the larger  
> project tree I don't care about or need.

When you say "checkout" do you really mean "git checkout" (IOW, you
still have the history for *everything* in .git but you just
don't want your working tree to be polluted by the stuff you don't
need at the moment) or are you talking about "partial cloning"?
The reason I'm asking is because a need for "partial cloning"
is very real around where I work. Which is easy to understand
because TeamWare (a precursor to BitKeeper) supports such a
concept. Now, before I get beaten to death: I DO understand why 
"partial cloning" and changeset-oriented SCMs don't go together
well. Yet, it seems to me that a sort of "transitive closure"
of the changesets for the files I'm actually interested in might
be a natural way of having such a feature.

>  But right now submodules are  
> the only way to approximate sparse checkout, 

For us submodules seem to be  the only way to approximate "sparse
cloning" and that's why the usage patterns I worry about don't
necessarily match the ones that drive "git submodule"

Thanks,
Roman.

Re: Making submodules easier to work with (auto-update on checkout or merge, stash & restore submodules)

[Ping Yin]

On Thu, May 1, 2008 at 4:19 AM, Tim Harper <timcharper@gmail.com> wrote:

>
>  I could see an issue if you changed the submodule version, and forgot to
> commit the version change in your super project, and then switched branches,
> and lost your version change.  Perhaps the correct behavior in this case
> would be to only automatically update the submodule version if it didn't
> differ from the superproject version in the "from" branch.
>

Agree. Doing this will make submodule be more consistent with the ordinary file.
Further, when switching to another branch with diffrent recorded
commit about the submodule and the submodule in the from branch has
been modified, just refuse to switch because this can be considered a
conflict and there is no easy way to merge.


-- 
Ping Yin