[RFC] submodule+shallow clone feature request

[RFC] submodule+shallow clone feature request

[Schuyler Duveen]

My use case is deploying from a git repository, which would be even more
graceful with the following features:

1. When 'git clone' has both --recursive and --depth, then submodules
are also checked out shallow (for speed/bandwidth).

2. Some way to specify an override on .gitmodules sources.  This is
because our .gitmodules includes public, read-only sources (github),
rather than our local repos we would prefer to deploy from (for the
purpose of reliability).

The other use-case for feature #2 is the read-only vs. writable
repository sources.  Developers that are also working on the submodules
should be able to clone from separate repository sources.  Though this
could be (and perhaps is) satisfied with a pushurl= in .gitmodules, I'd
like to keep the push url non-public, and that still would leave us
unable to deploy from different urls.

For #2 maybe something like this:

$ git clone --recursive --depth 1 --modules foo.modules\
> git@example.com/foo.git
where foo.modules contains:
<begin>
[submodule "bar"]
	path = bar
	url = git@example.com/bar.git
[submodule "bar/baz"]
	path = bar/baz
	url = git@example.com/baz.git
<end>

Notice how the .gitmodules needs to be able to specify recursive sources
as well.  Presumably `git submodule init` would take --modules argument
as well (and maybe git submodule update, too).

I'm not well-versed in the git source code yet, but poking around
suggests that this is doable.  Would a patch be well received (or does
someone want to do it for me :-)

cheers,
sky

Re: [RFC] submodule+shallow clone feature request

[Avery Pennarun]

On Wed, Feb 10, 2010 at 4:39 PM, Schuyler Duveen <sky@columbia.edu> wrote:
> 2. Some way to specify an override on .gitmodules sources.  This is
> because our .gitmodules includes public, read-only sources (github),
> rather than our local repos we would prefer to deploy from (for the
> purpose of reliability).

You can already override this by editing .git/config after running
'git submodule init'.  Of course, doing so precludes using --recursive
in "git clone."

> I'm not well-versed in the git source code yet, but poking around
> suggests that this is doable.  Would a patch be well received (or does
> someone want to do it for me :-)

The git-submodule stuff is rarely maintained.  I expect patches that
improve it will meet with generally positive response, but it's
somewhat unlikely that anyone will write them for you.

Have fun,

Avery

Re: [RFC] submodule+shallow clone feature request

[Junio C Hamano]

Schuyler Duveen <sky@columbia.edu> writes:

> My use case is deploying from a git repository, which would be even more
> graceful with the following features:
>
> 1. When 'git clone' has both --recursive and --depth, then submodules
> are also checked out shallow (for speed/bandwidth).
>
> 2. Some way to specify an override on .gitmodules sources.  This is
> because our .gitmodules includes public, read-only sources (github),
> rather than our local repos we would prefer to deploy from (for the
> purpose of reliability).

These should be doable if you do not use --recursive, so I don't think
they are insurmountable issues.  I suspect many people would welcome such
enhancements to the "git submodule" potty.

I think the current implementation of "--recursive" is an attempt to help
people with a particular view (all submodules are interesting and the user
needs an access to them immediately) while not trying to help others with
different needs.  Contribution from the latter class of users to change
the situation would be good.

Re: [RFC] submodule+shallow clone feature request

[Johannes Schindelin]

Hi,

On Wed, 10 Feb 2010, Junio C Hamano wrote:

> Schuyler Duveen <sky@columbia.edu> writes:
> 
> > My use case is deploying from a git repository, which would be even more
> > graceful with the following features:
> >
> > 1. When 'git clone' has both --recursive and --depth, then submodules
> > are also checked out shallow (for speed/bandwidth).
> >
> > 2. Some way to specify an override on .gitmodules sources.  This is
> > because our .gitmodules includes public, read-only sources (github),
> > rather than our local repos we would prefer to deploy from (for the
> > purpose of reliability).
> 
> These should be doable if you do not use --recursive, so I don't think 
> they are insurmountable issues.  I suspect many people would welcome 
> such enhancements to the "git submodule" potty.

Yes. Note, though, that the problems of enhancing git-submodule are not 
technical, as we can learn from the recent history, including the lack of 
support for rebasing submodules (there _were_ patches!).

Ciao,
Dscho

Re: [RFC] submodule+shallow clone feature request

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Yes. Note, though, that the problems of enhancing git-submodule are not 
> technical, as we can learn from the recent history, including the lack of 
> support for rebasing submodules (there _were_ patches!).

Sorry I don't recall.  Were they of 'next' quality?  How well were they
reviewed?

Re: [RFC] submodule+shallow clone feature request

[Johannes Schindelin]

Hi,

On Wed, 10 Feb 2010, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > Yes. Note, though, that the problems of enhancing git-submodule are 
> > not technical, as we can learn from the recent history, including the 
> > lack of support for rebasing submodules (there _were_ patches!).
> 
> Sorry I don't recall.  Were they of 'next' quality?  How well were they 
> reviewed?

Obviously not, otherwise you would have applied them, no?

OTOH I found the technical details rather trivial, so maybe they were 
'next' quality, but there was another reason you did not apply them.

I just know that from my daily workflow, I deeply miss rebasing 
submodules. But then, I listed the issues I have with submodules, and this 
list was welcomed with unbelievable enthusiasm.

Ciao,
Dscho

Re* [RFC] submodule+shallow clone feature request

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Wed, 10 Feb 2010, Junio C Hamano wrote:
>
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>> 
>> > Yes. Note, though, that the problems of enhancing git-submodule are 
>> > not technical, as we can learn from the recent history, including the 
>> > lack of support for rebasing submodules (there _were_ patches!).
>> 
>> Sorry I don't recall.  Were they of 'next' quality?  How well were they 
>> reviewed?
>
> Obviously not, otherwise you would have applied them, no?
>
> OTOH I found the technical details rather trivial, so maybe they were 
> 'next' quality, but there was another reason you did not apply them.

Well, I spent some time digging the mail archive myself.  I think you were
talking about this thread:

    http://thread.gmane.org/gmane.comp.version-control.git/116918

If this is not the thread you were talking about, please discard/disregard
the remainder of this message, and give me a better pointer instead.

The thread ends with you asking me:

    Junio, how about it? post 1.6.3 or not?  It is a well contained change, 
    with little chance of breaking something, but offers a more sensible 
    workflow here.

and I said:

    I am afraid it is a bit too late for "improvements" after -rc1.
    People survived without the new feature until now, and they can wait a
    bit longer for the next one....

Obviously, after that nothing happened.  We have some people who send
reminders for good topics after the original thread died without producing
a visible result.  I'd ask you to do the same (when you can---as everybody
else, you don't work full time on git; neither do I).

An honest answer to my question would have been "Yes, I reviewed it and
mentored the original author through a few cycles of revisions.  After
that process the patch was of 'next' quality.  This however happened
during the pre-release freeze, and unfortunately nobody (including you or
I) rememebered to bring up the issue again after the release to move
things forward.  It fell through cracks."

Instead, what you wrote makes it sounds as if I blocked a technically
correct solution to a real problem on some political grounds, doesn't it?
What effect did you hope your insinuation to have on readers of this list?

It does not help to give a new person a false impression that it takes
more than producing a good solution to a real problem to contribute to the
project.  Especially, the "falling through the crack" didn't have anything
to do with it being submodule changes.  Please stop spreading FUD and try
to be constructive as others do.

To restart the discussion so that we can have it (if it is a good change)
after 1.7.0 ships, here is a pointer to the last revision of the patch.

    http://article.gmane.org/gmane.comp.version-control.git/117394/raw

I didn't check if it needs rebasing to the recent codebase, but I expect
people who work with submodules in real life (I don't count) to comment on
it, and re-polish it into an applicable shape if necessary, by the time
post 1.7.0 cycle opens.

-- >8 --
From: Peter Hutterer <peter.hutterer@who-t.net>
Subject: Re: [PATCH/RESEND] git-submodule: add support for --rebase.
Date: Fri, 24 Apr 2009 09:06:38 +1000

'git submodule update --rebase' rebases your local branch on top of what
would have been checked out to a detached HEAD otherwise.

In some cases, detaching the HEAD when updating a submodule complicates
the workflow to commit to this submodule (checkout master, rebase, then
commit).  For submodules that require frequent updates but infrequent
(if any) commits, a rebase can be executed directly by the git-submodule
command, ensuring that the submodules stay on their respective branches.

git-config key: submodule.$name.rebase (bool)

Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---

> With this squashed in, I actually would not be too opposed to put this 
> into 1.6.3, as it _is_ an improvement.
> 
> -- snipsnap --
> [PATCH] To be squashed in
> 
> This does three things:
> 
> - add .gitmodules support for rebase,
> - use --bool for the git config calls for type checking, and
> - rename ambiguous t7404 to t7406.
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Squashed in, thank you. One typo fixed in last testcase (does-not-mater ->
does-not-matter).

 Documentation/git-submodule.txt |   14 ++++-
 Documentation/gitmodules.txt    |    3 +
 git-submodule.sh                |   33 ++++++++-
 t/t7406-submodule-update.sh     |  140 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 184 insertions(+), 6 deletions(-)
 create mode 100755 t/t7406-submodule-update.sh

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 3b8df44..0286409 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -12,7 +12,7 @@ SYNOPSIS
 'git submodule' [--quiet] add [-b branch] [--] <repository> <path>
 'git submodule' [--quiet] status [--cached] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
-'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--] [<path>...]
+'git submodule' [--quiet] update [--init] [-N|--no-fetch] [--rebase] [--] [<path>...]
 'git submodule' [--quiet] summary [--summary-limit <n>] [commit] [--] [<path>...]
 'git submodule' [--quiet] foreach <command>
 'git submodule' [--quiet] sync [--] [<path>...]
@@ -113,7 +113,8 @@ init::
 update::
 	Update the registered submodules, i.e. clone missing submodules and
 	checkout the commit specified in the index of the containing repository.
-	This will make the submodules HEAD be detached.
+	This will make the submodules HEAD be detached unless '--rebase' is
+	specified or the key `submodule.$name.rebase` is set to `true`.
 +
 If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
@@ -177,6 +178,15 @@ OPTIONS
 	This option is only valid for the update command.
 	Don't fetch new objects from the remote site.
 
+--rebase::
+	This option is only valid for the update command.
+	Rebase the current branch onto the commit recorded in the
+	superproject. If this option is given, the submodule's HEAD will not
+	be detached. If a a merge failure prevents this process, you will have
+	to resolve these failures with linkgit:git-rebase[1].
+	If the key `submodule.$name.rebase` is set to `true`, this option is
+	implicit.
+
 <path>...::
 	Paths to submodule(s). When specified this will restrict the command
 	to only operate on the submodules found at the specified paths.
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index d1a17e2..7c22c40 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -30,6 +30,9 @@ submodule.<name>.path::
 submodule.<name>.url::
 	Defines an url from where the submodule repository can be cloned.
 
+submodule.<name>.rebase::
+	Defines that the submodule should be rebased by default.
+
 
 EXAMPLES
 --------
diff --git a/git-submodule.sh b/git-submodule.sh
index 7c2e060..b7c9bdc 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -17,6 +17,7 @@ branch=
 quiet=
 cached=
 nofetch=
+rebase=
 
 #
 # print stuff on stdout unless -q was specified
@@ -287,6 +288,11 @@ cmd_init()
 		git config submodule."$name".url "$url" ||
 		die "Failed to register url for submodule path '$path'"
 
+		test true != "$(git config -f .gitmodules --bool \
+			submodule."$name".rebase)" ||
+		git config submodule."$name".rebase true ||
+		die "Failed to register submodule path '$path' as rebasing"
+
 		say "Submodule '$name' ($url) registered for path '$path'"
 	done
 }
@@ -314,6 +320,10 @@ cmd_update()
 			shift
 			nofetch=1
 			;;
+		-r|--rebase)
+			shift
+			rebase=true
+			;;
 		--)
 			shift
 			break
@@ -332,6 +342,7 @@ cmd_update()
 	do
 		name=$(module_name "$path") || exit
 		url=$(git config submodule."$name".url)
+		rebase_module=$(git config --bool submodule."$name".rebase)
 		if test -z "$url"
 		then
 			# Only mention uninitialized submodules when its
@@ -352,6 +363,11 @@ cmd_update()
 			die "Unable to find current revision in submodule path '$path'"
 		fi
 
+		if test true = "$rebase"
+		then
+			rebase_module=true
+		fi
+
 		if test "$subsha1" != "$sha1"
 		then
 			force=
@@ -367,11 +383,20 @@ cmd_update()
 				die "Unable to fetch in submodule path '$path'"
 			fi
 
-			(unset GIT_DIR; cd "$path" &&
-				  git-checkout $force -q "$sha1") ||
-			die "Unable to checkout '$sha1' in submodule path '$path'"
+			if test true = "$rebase_module"
+			then
+				command="git-rebase"
+				action="rebase"
+				msg="rebased onto"
+			else
+				command="git-checkout $force -q"
+				action="checkout"
+				msg="checked out"
+			fi
 
-			say "Submodule path '$path': checked out '$sha1'"
+			(unset GIT_DIR; cd "$path" && $command "$sha1") ||
+			die "Unable to $action '$sha1' in submodule path '$path'"
+			say "Submodule path '$path': $msg '$sha1'"
 		fi
 	done
 }
diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
new file mode 100755
index 0000000..3442c05
--- /dev/null
+++ b/t/t7406-submodule-update.sh
@@ -0,0 +1,140 @@
+#!/bin/sh
+#
+# Copyright (c) 2009 Red Hat, Inc.
+#
+
+test_description='Test updating submodules
+
+This test verifies that "git submodule update" detaches the HEAD of the
+submodule and "git submodule update --rebase" does not detach the HEAD.
+'
+
+. ./test-lib.sh
+
+
+compare_head()
+{
+    sha_master=`git-rev-list --max-count=1 master`
+    sha_head=`git-rev-list --max-count=1 HEAD`
+
+    test "$sha_master" = "$sha_head"
+}
+
+
+test_expect_success 'setup a submodule tree' '
+	echo file > file &&
+	git add file &&
+	test_tick &&
+	git commit -m upstream
+	git clone . super &&
+	git clone super submodule &&
+	(cd super &&
+	 git submodule add ../submodule submodule &&
+	 test_tick &&
+	 git commit -m "submodule" &&
+	 git submodule init submodule
+	) &&
+	(cd submodule &&
+	echo "line2" > file &&
+	git add file &&
+	git commit -m "Commit 2"
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  git pull --rebase origin
+	 ) &&
+	 git add submodule &&
+	 git commit -m "submodule update"
+	)
+'
+
+test_expect_success 'submodule update detaching the HEAD ' '
+	(cd super/submodule &&
+	 git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update submodule &&
+	 cd submodule &&
+	 ! compare_head
+	)
+'
+
+test_expect_success 'submodule update --rebase staying on master' '
+	(cd super/submodule &&
+	  git checkout master
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update --rebase submodule &&
+	 cd submodule &&
+	 compare_head
+	)
+'
+
+test_expect_success 'submodule update - rebase true in .git/config' '
+	(cd super &&
+	 git config submodule.submodule.rebase true
+	) &&
+	(cd super/submodule &&
+	  git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update submodule &&
+	 cd submodule &&
+	 compare_head
+	)
+'
+
+test_expect_success 'submodule update - rebase false in .git/config but --rebase given' '
+	(cd super &&
+	 git config submodule.submodule.rebase false
+	) &&
+	(cd super/submodule &&
+	  git reset --hard HEAD~1
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update --rebase submodule &&
+	 cd submodule &&
+	 compare_head
+	)
+'
+
+test_expect_success 'submodule update - rebase false in .git/config' '
+	(cd super &&
+	 git config submodule.submodule.rebase false
+	) &&
+	(cd super/submodule &&
+	  git reset --hard HEAD^
+	) &&
+	(cd super &&
+	 (cd submodule &&
+	  compare_head
+	 ) &&
+	 git submodule update submodule &&
+	 cd submodule &&
+	 ! compare_head
+	)
+'
+
+test_expect_success 'submodule init picks up rebase' '
+	(cd super &&
+	 git config submodule.rebasing.url git://non-existing/git &&
+	 git config submodule.rebasing.path does-not-matter &&
+	 git config submodule.rebasing.rebase true &&
+	 git submodule init rebasing &&
+	 test true = $(git config --bool submodule.rebasing.rebase)
+	)
+'
+
+test_done
-- 
1.6.3.rc1.2.gfa66a

Re: Re* [RFC] submodule+shallow clone feature request

[Peter Hutterer]

On Wed, Feb 10, 2010 at 10:19:42PM -0800, Junio C Hamano wrote:
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > On Wed, 10 Feb 2010, Junio C Hamano wrote:
> >
> >> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> >> 
> >> > Yes. Note, though, that the problems of enhancing git-submodule are 
> >> > not technical, as we can learn from the recent history, including the 
> >> > lack of support for rebasing submodules (there _were_ patches!).
> >> 
> >> Sorry I don't recall.  Were they of 'next' quality?  How well were they 
> >> reviewed?
> >
> > Obviously not, otherwise you would have applied them, no?
> >
> > OTOH I found the technical details rather trivial, so maybe they were 
> > 'next' quality, but there was another reason you did not apply them.
> 
> Well, I spent some time digging the mail archive myself.  I think you were
> talking about this thread:
> 
>     http://thread.gmane.org/gmane.comp.version-control.git/116918
> 
> If this is not the thread you were talking about, please discard/disregard
> the remainder of this message, and give me a better pointer instead.
> 
> The thread ends with you asking me:
> 
>     Junio, how about it? post 1.6.3 or not?  It is a well contained change, 
>     with little chance of breaking something, but offers a more sensible 
>     workflow here.
> 
> and I said:
> 
>     I am afraid it is a bit too late for "improvements" after -rc1.
>     People survived without the new feature until now, and they can wait a
>     bit longer for the next one....
> 
> Obviously, after that nothing happened.  We have some people who send
> reminders for good topics after the original thread died without producing
> a visible result.  I'd ask you to do the same (when you can---as everybody
> else, you don't work full time on git; neither do I).

[...]

> To restart the discussion so that we can have it (if it is a good change)
> after 1.7.0 ships, here is a pointer to the last revision of the patch.
> 
>     http://article.gmane.org/gmane.comp.version-control.git/117394/raw

Thanks for CCing me, I'm not on the list at the moment.
FWIW, Johannes pinged me about this patch a few weeks after that, but after
revisiting it a few times I found some issues with it. Here's the email I
sent to Johannes on April 24, my apologies that this was a private email
only and did not reach the list.

"Sorry about the delay again, I've been a bit busy lately.
I've thought about it a bit more and tbh. I don't think this patch - even if
rebased - should be merged.

The original idea was that a module can be marked for auto-rebasing in
.gitmodules. The issue with that is that AFAIK git submodule does not store
branch info. So such auto-rebasing would only work provided it would be on
the master branch. Anything else would require a fancier script than my
patch including specifying which branch should be checked out in the
original clone.

Right now, I don't have the time to design such a patch and I'm not even
sure how much it is needed.
With git submodule foreach it's relatively simple to just do the auto-rebase
setting for all modules and I would not be surprised if the majority of
use-cases require auto-rebasing on all modules anyway.

Does that make sense?
"

So in this particular case the patchset was withdrawn by me for technical
reasons (and the lack of time to sort out the details). It should have
communicated better - again, my apologies for that.

Cheers,
  Peter