RFC: Between git-subtree and git-submodules

RFC: Between git-subtree and git-submodules

[Santi Béjar]

[-- Attachment #1: Type: text/plain, Size: 4502 bytes --]

Hi *,

  now that there is a lot of traffic about git-subtree and git-submodules I
  would like to show you why *I* don't use neither of them, but something in
  between.

  First my requirements:

1) Everything[1] must be available from the same repository/branch (so I'm not
   worried about repository size)
2) The history must be as clean as possible
3) The directory content must be equal to the external module, at least when
   you add/update it[2]
4) The external module should be able to switch back and forth between
   different versions.

[1] Everything means all that you need to checkout all the commits in the
superproject not in the submodule.
[2] A consequence of 3) is that I lose all
change I've made in the subdirectory, if they are important I have to extract
them, apply them and add the module back.

git-submodule is rule out because of 1) but accomplish 2), 3) and
4). git-subtree is rule out because of 2) (even with --squash) 3) and 4)
without --squash but accomplish 1) and 4) with --squash. So I need something
in between or a mixture of both.

At the end I've done what I called originally git-subtree, but now I've written
the prune mode of git-subtree.

The idea is that you want to add the content of a module in a subdirectory and
that's all! I think that this simplicity is also very powerful as it
is very clear how it
behaves. You just get a commit each time you add a subtree (normal commit
not a merge) without the history of the subtree.

You get something like this:

$ git log --graph
* ee225bd Subtree 'a/': 895916a Commit message 1
* aa345dg Modification to a/Makefile
* ddcd676 Subtree 'a/': 9a053f2 Commit message 2
* ea35faf Indent, whitespaces,...

Later you can extract/split all the modifications to a/ with "git-subtree
split" (not yet implemented).

If you merge two branches with different content in the subtree you just merge
them the normal way as with any other file.

It works quite well in my use case.

Below you can find the patch to git-subtree to add this prune mode (also
attatched because of whitespace corruption). Only slightly tested as I
normally use
my git-subtree command (completely different) and I've just patch git-subtree
but you can get the idea of what it does.

HTH,
Santi

diff --git i/git-subtree.sh w/git-subtree.sh
index 781eef3..7706d72 100755
--- i/git-subtree.sh
+++ w/git-subtree.sh
@@ -27,6 +27,7 @@ onto=         try connecting new tree to an existing one
 rejoin        merge the new branch back into HEAD
  options for 'add', 'merge', 'pull' and 'push'
 squash        merge subtree changes as a single commit
+prune         prune history
 "
 eval $(echo "$OPTS_SPEC" | git rev-parse --parseopt -- "$@" || echo exit $?)

@@ -44,6 +45,7 @@ rejoin=
 ignore_joins=
 annotate=
 squash=
+prune=
 message=

 debug()
@@ -92,6 +94,8 @@ while [ $# -gt 0 ]; do
 		--no-ignore-joins) ignore_joins= ;;
 		--squash) squash=1 ;;
 		--no-squash) squash= ;;
+		--prune) prune=1;;
+		--no-prune) prune=;;
 		--) break ;;
 		*) die "Unexpected option: $opt" ;;
 	esac
@@ -110,7 +114,7 @@ if [ -z "$prefix" ]; then
 fi

 case "$command" in
-	add) [ -e "$prefix" ] &&
+	add) [ -e "$prefix" -a -z "$prune" ] &&
 		die "prefix '$prefix' already exists." ;;
 	*)   [ -e "$prefix" ] ||
 		die "'$prefix' does not exist; use 'git subtree add'" ;;
@@ -359,6 +363,17 @@ squash_msg()
 	echo "git-subtree-split: $newsub"
 }

+prune_msg()
+{
+	dir="$1"
+	newsub="$2"
+	
+	git show -s --pretty="tformat:Subtree '$dir/': %h %s" $newsub
+	echo
+	echo "git-subtree-dir: $dir"
+	echo "git-subtree-split: $newsub"
+}
+
 toptree_for_commit()
 {
 	commit="$1"
@@ -464,7 +479,7 @@ ensure_clean()

 cmd_add()
 {
-	if [ -e "$dir" ]; then
+	if [ -e "$dir" -a -z "$prune" ]; then
 		die "'$dir' already exists.  Cannot add."
 	fi

@@ -498,6 +513,10 @@ cmd_add_commit()
 	rev="$1"
 	
 	debug "Adding $dir as '$rev'..."
+	if [ -d "$dir" ]; then
+	    #TODO: write it with plumbing commands
+	    git rm -r -q $dir
+	fi
 	git read-tree --prefix="$dir" $rev || exit $?
 	git checkout -- "$dir" || exit $?
 	tree=$(git write-tree) || exit $?
@@ -513,6 +532,9 @@ cmd_add_commit()
 		rev=$(new_squash_commit "" "" "$rev") || exit $?
 		commit=$(add_squashed_msg "$rev" "$dir" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?
+	elif [ -n "$prune" ]; then
+		commit=$(prune_msg "$dir" "$rev" |
+			 git commit-tree $tree -p $headrev) || exit $?
 	else
 		commit=$(add_msg "$dir" "$headrev" "$rev" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?

[-- Attachment #2: git-subtree-prune-mode.patch --]
[-- Type: text/x-diff, Size: 2257 bytes --]

diff --git i/git-subtree.sh w/git-subtree.sh
index 781eef3..7706d72 100755
--- i/git-subtree.sh
+++ w/git-subtree.sh
@@ -27,6 +27,7 @@ onto=         try connecting new tree to an existing one
 rejoin        merge the new branch back into HEAD
  options for 'add', 'merge', 'pull' and 'push'
 squash        merge subtree changes as a single commit
+prune         prune history
 "
 eval $(echo "$OPTS_SPEC" | git rev-parse --parseopt -- "$@" || echo exit $?)
 
@@ -44,6 +45,7 @@ rejoin=
 ignore_joins=
 annotate=
 squash=
+prune=
 message=
 
 debug()
@@ -92,6 +94,8 @@ while [ $# -gt 0 ]; do
 		--no-ignore-joins) ignore_joins= ;;
 		--squash) squash=1 ;;
 		--no-squash) squash= ;;
+		--prune) prune=1;;
+		--no-prune) prune=;;
 		--) break ;;
 		*) die "Unexpected option: $opt" ;;
 	esac
@@ -110,7 +114,7 @@ if [ -z "$prefix" ]; then
 fi
 
 case "$command" in
-	add) [ -e "$prefix" ] && 
+	add) [ -e "$prefix" -a -z "$prune" ] &&
 		die "prefix '$prefix' already exists." ;;
 	*)   [ -e "$prefix" ] || 
 		die "'$prefix' does not exist; use 'git subtree add'" ;;
@@ -359,6 +363,17 @@ squash_msg()
 	echo "git-subtree-split: $newsub"
 }
 
+prune_msg()
+{
+	dir="$1"
+	newsub="$2"
+	
+	git show -s --pretty="tformat:Subtree '$dir/': %h %s" $newsub
+	echo
+	echo "git-subtree-dir: $dir"
+	echo "git-subtree-split: $newsub"
+}
+
 toptree_for_commit()
 {
 	commit="$1"
@@ -464,7 +479,7 @@ ensure_clean()
 
 cmd_add()
 {
-	if [ -e "$dir" ]; then
+	if [ -e "$dir" -a -z "$prune" ]; then
 		die "'$dir' already exists.  Cannot add."
 	fi
 
@@ -498,6 +513,10 @@ cmd_add_commit()
 	rev="$1"
 	
 	debug "Adding $dir as '$rev'..."
+	if [ -d "$dir" ]; then
+	    #TODO: write it with plumbing commands
+	    git rm -r -q $dir
+	fi
 	git read-tree --prefix="$dir" $rev || exit $?
 	git checkout -- "$dir" || exit $?
 	tree=$(git write-tree) || exit $?
@@ -513,6 +532,9 @@ cmd_add_commit()
 		rev=$(new_squash_commit "" "" "$rev") || exit $?
 		commit=$(add_squashed_msg "$rev" "$dir" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?
+	elif [ -n "$prune" ]; then
+		commit=$(prune_msg "$dir" "$rev" |
+			 git commit-tree $tree -p $headrev) || exit $?
 	else
 		commit=$(add_msg "$dir" "$headrev" "$rev" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?

Re: RFC: Between git-subtree and git-submodules

[Avery Pennarun]

On Fri, Jul 23, 2010 at 8:13 PM, Santi Béjar <santi@agolina.net> wrote:
>  First my requirements:
>
> 1) Everything[a] must be available from the same repository/branch (so I'm not
>   worried about repository size)
> 2) The history must be as clean as possible
> 3) The directory content must be equal to the external module, at least when
>   you add/update it[b]
> 4) The external module should be able to switch back and forth between
>   different versions.
>
> [a] Everything means all that you need to checkout all the commits in the
> superproject not in the submodule.
> [b] A consequence of #3 is that I lose all
> change I've made in the subdirectory, if they are important I have to extract
> them, apply them and add the module back.
>
> git-submodule is rule out because of #1 but accomplish #2, #3 and
> #4. git-subtree is rule out because of #2 (even with --squash).
> [It fails at] #3 and #4
> without --squash but accomplish #1 and #4 with --squash. So I need something
> in between or a mixture of both.

I admit to having had some trouble parsing the above, so I moved some
punctuation marks around.  Please let me know if I've made a mistake.

If I understand correctly, you're claiming (indirectly) that
git-subtree without --squash does not accomplish #1.  I don't see how
this is the case.  Am I misreading?  I think git-subtree accomplishes
#1 in both modes.

I don't understand what you mean when you say (#2) git-subtree doesn't
keep your history "as clean as possible."  What is "as clean as
possible" and what part of git-subtree's history results don't you
like?  (Of course it's very different with and without --squash.)

With #3, I can see that you want something different than I do; you
want to silently revert your own patches out of the submodule's
history, when you upgrade the submodule to a new version.  Personally,
I find this concept a bit objectionable (it's like "git merge -s
ours"), but okay, it's pretty easy to implement, and you've submitted
a patch to git-subtree that does it.  My question is: why would you
want this?  Isn't it clearer to 'git revert' the patches you don't
want?

And for #4, it's true that git-subtree without --squash does not allow
you to easily rewind to an older version of the submodule, while with
--squash it does.

It sounds to me like, if we added your patch to git-subtree, then
git-subtree --squash would solve #1, #3, and #4.  And maybe we could
fix #2 as well. Correct?

Thanks,

Avery

Re: RFC: Between git-subtree and git-submodules

[Santi Béjar]

On Sat, Jul 24, 2010 at 2:57 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Fri, Jul 23, 2010 at 8:13 PM, Santi Béjar <santi@agolina.net> wrote:
>>  First my requirements:
>>
>> 1) Everything[a] must be available from the same repository/branch (so I'm not
>>   worried about repository size)
>> 2) The history must be as clean as possible
>> 3) The directory content must be equal to the external module, at least when
>>   you add/update it[b]
>> 4) The external module should be able to switch back and forth between
>>   different versions.
>>
>> [a] Everything means all that you need to checkout all the commits in the
>> superproject not in the submodule.
>> [b] A consequence of #3 is that I lose all
>> change I've made in the subdirectory, if they are important I have to extract
>> them, apply them and add the module back.
>>
>> git-submodule is rule out because of #1 but accomplish #2, #3 and
>> #4. git-subtree is rule out because of #2 (even with --squash).
>> [It fails at] #3 and #4
>> without --squash but accomplish #1 and #4 with --squash. So I need something
>> in between or a mixture of both.
>
> I admit to having had some trouble parsing the above, so I moved some
> punctuation marks around.  Please let me know if I've made a mistake.
>
> If I understand correctly, you're claiming (indirectly) that
> git-subtree without --squash does not accomplish #1.  I don't see how
> this is the case.  Am I misreading?  I think git-subtree accomplishes
> #1 in both modes.

Yes, git-subtree accomplishes #1 in both modes. --squash only applies to #4.

>
> I don't understand what you mean when you say (#2) git-subtree doesn't
> keep your history "as clean as possible."  What is "as clean as
> possible" and what part of git-subtree's history results don't you
> like?  (Of course it's very different with and without --squash.)

With git-subtree you always have the subtree history (even if it is
squashed). So when you merge a second time the submodule you get the always
the history of the subtree (even with --squash). So you basically always have
at least two branches while examining the history. Compare this
squashed history:

$ git log --graph --oneline
*   bb2dc25 (HEAD, master) Merge commit
'08b917ee90ecfd7b666364fe4ebb92aee5cdd2f7'
|\
| * 08b917e Squashed 'latex/' changes from ea35faf..895916a
* |   9de91f1 Merge commit 'b1b4c36bb8358582a6a20bb500bf98421428e2ca' as 'latex'
|\|
| * b1b4c36 Squashed 'latex/' content from commit ea35faf
* ea35faf Indent, whitespaces,...

with this pruned history:

$ git log --graph --oneline
* 8703aec (HEAD, master) Subtree 'latex/': 895916a Add files subcommand
* a942284 Subtree 'latex/': ea35faf Indent, whitespaces,...
* ea35faf Indent, whitespaces,...

But I understand that it can only be this way because of #3.

>
> With #3, I can see that you want something different than I do; you
> want to silently revert your own patches out of the submodule's
> history, when you upgrade the submodule to a new version.  Personally,
> I find this concept a bit objectionable (it's like "git merge -s
> ours"), but okay, it's pretty easy to implement, and you've submitted
> a patch to git-subtree that does it.  My question is: why would you
> want this?  Isn't it clearer to 'git revert' the patches you don't
> want?

I prefer to do all the modifications in an external repository, even if at the
end it is really a fork of the upstream repository. I think the proper place
to modify the files in a subtree is in an external repository.

>
> And for #4, it's true that git-subtree without --squash does not allow
> you to easily rewind to an older version of the submodule, while with
> --squash it does.
>
> It sounds to me like, if we added your patch to git-subtree, then
> git-subtree --squash would solve #1, #3, and #4.  And maybe we could
> fix #2 as well. Correct?

My patch does not change the behavior of --squash, it adds --prune.

Thanks,
Santi

Re: RFC: Between git-subtree and git-submodules

[Santi Béjar]

> With #3, I can see that you want something different than I do; you
> want to silently revert your own patches out of the submodule's
> history, when you upgrade the submodule to a new version.  Personally,
> I find this concept a bit objectionable (it's like "git merge -s
> ours"), but okay, it's pretty easy to implement, and you've submitted
> a patch to git-subtree that does it.  My question is: why would you
> want this?  Isn't it clearer to 'git revert' the patches you don't
> want?

While writing the docs for the --prune option I've found this sentence in
git-subtree.txt:

	'git subtree merge --squash'
	always adjusts the subtree to match the exactly
	specified commit, even if getting to that commit would
	require undoing some changes that were added earlier.

I think this is not true, and it contradicts with what you've just said. Or I
don't understand it correctly.

Thanks,
Santi

Re: RFC: Between git-subtree and git-submodules

[Avery Pennarun]

On Sat, Jul 24, 2010 at 8:28 PM, Santi Béjar <santi@agolina.net> wrote:
>> With #3, I can see that you want something different than I do; you
>> want to silently revert your own patches out of the submodule's
>> history, when you upgrade the submodule to a new version.  Personally,
>> I find this concept a bit objectionable (it's like "git merge -s
>> ours"), but okay, it's pretty easy to implement, and you've submitted
>> a patch to git-subtree that does it.  My question is: why would you
>> want this?  Isn't it clearer to 'git revert' the patches you don't
>> want?
>
> While writing the docs for the --prune option I've found this sentence in
> git-subtree.txt:
>
>        'git subtree merge --squash'
>        always adjusts the subtree to match the exactly
>        specified commit, even if getting to that commit would
>        require undoing some changes that were added earlier.
>
> I think this is not true, and it contradicts with what you've just said. Or I
> don't understand it correctly.

I agree that this is a bit unclear.  Basically it rewinds the *base*
of the subtree to match the commit you specify.  Patches you've
applied locally are not removed.

What it actually does is it creates a new commit that applies the
following patch:

    git diff oldbase newbase

If you've made any of your own patches that conflict with this, it'll
generate the conflict and then stop.  Otherwise it'll merge in your
changes as you might expect.

Have fun,

Avery