[PATCH] Pass --upload-pack and --receive-pack through submodules.

[PATCH] Pass --upload-pack and --receive-pack through submodules.

[Jason Riedy]

While I no longer have to worry about a zillion ancient OS versions,
I now have to worry about a remote site where I cannot control the
path for non-interactive shells.  Thus, submodules need to handle
explicitly specified git-upload-pack and git-receive-pack programs.

There may be a fun shell quoting dance to avoid the extra conditionals.
Any such dance would be even more confusing to read.

Signed-off-by: Jason Riedy <jason@acm.org>
---
 git-submodule.sh             |   81 +++++++++++++++++++++++++++++++++++++++--
 t/t7404-submodule-packbin.sh |   53 +++++++++++++++++++++++++++
 2 files changed, 130 insertions(+), 4 deletions(-)
 create mode 100755 t/t7404-submodule-packbin.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index 2f47e06..1a8a968 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -90,6 +90,8 @@ module_clone()
 {
 	path=$1
 	url=$2
+	uploadpack=$3
+	receivepack=$4
 
 	# If there already is a directory at the submodule path,
 	# expect it to be empty (since that is the default checkout
@@ -105,8 +107,23 @@ module_clone()
 	test -e "$path" &&
 	die "A file already exist at path '$path'"
 
-	git-clone -n "$url" "$path" ||
+	if test "$uploadpack"
+	then
+	    git-clone --upload-pack $uploadpack -n "$url" "$path"
+	else
+	    git-clone -n "$url" "$path"
+	fi ||
 	die "Clone of '$url' into submodule path '$path' failed"
+	if test "$uploadpack"
+	then
+	    git config -f "${path}/.git/config" remote.origin.uploadpack "$uploadpack" ||
+	    echo "  Warn: Failed to set uploadpack for '$url' in submodule path '$path'."
+	fi
+	if test "$receivepack"
+	then
+	    git config -f "${path}/.git/config" remote.origin.receivepack "$receivepack" ||
+	    echo "  Warn: Failed to set receivepack for '$url' in submodule path '$path'."
+	fi
 }
 
 #
@@ -130,6 +147,16 @@ cmd_add()
 		-q|--quiet)
 			quiet=1
 			;;
+		-u|--upload-pack)
+			case "$2" in '') usage ;; esac
+			uploadpack=$2
+			shift
+			;;
+		--receive-pack)
+			case "$2" in '') usage ;; esac
+			receivepack=$2
+			shift
+			;;
 		--)
 			shift
 			break
@@ -191,9 +218,17 @@ cmd_add()
 			;;
 		esac
 		git config submodule."$path".url "$url"
+		if test "$uploadpack"
+		then
+		    git config submodule."$path".uploadpack "$uploadpack"
+		fi
+		if test "$receivepack"
+		then
+		    git config submodule."$path".receivepack "$receivepack"
+		fi
 	else
 
-		module_clone "$path" "$realrepo" || exit
+		module_clone "$path" "$realrepo" "$uploadpack" "$receivepack" || exit
 		(unset GIT_DIR; cd "$path" && git checkout -f -q ${branch:+-b "$branch" "origin/$branch"}) ||
 		die "Unable to checkout submodule '$path'"
 	fi
@@ -202,7 +237,19 @@ cmd_add()
 	die "Failed to add submodule '$path'"
 
 	git config -f .gitmodules submodule."$path".path "$path" &&
-	git config -f .gitmodules submodule."$path".url "$repo" &&
+	git config -f .gitmodules submodule."$path".url "$repo" ||
+	die "Failed to register submodule '$path'"
+
+	if test "$uploadpack"
+        then
+	    git config -f .gitmodules submodule."$path".uploadpack "$uploadpack" ||
+	    die "Failed to register submodule '$path'"
+	fi
+	if test "$receivepack"
+        then
+	    git config -f .gitmodules submodule."$path".receivepack "$receivepack" ||
+	    die "Failed to register submodule '$path'"
+	fi
 	git add .gitmodules ||
 	die "Failed to register submodule '$path'"
 }
@@ -277,6 +324,19 @@ cmd_init()
 		git config submodule."$name".url "$url" ||
 		die "Failed to register url for submodule path '$path'"
 
+		uploadpack=$(git config -f .gitmodules submodule."$name".uploadpack)
+		receivepack=$(git config -f .gitmodules submodule."$name".receivepack)
+		if test "$uploadpack"
+		then
+		    git config submodule."$name".uploadpack "$uploadpack" ||
+		    echo "  Warn: Failed to set uploadpack for '$url' in submodule path '$path'."
+		fi
+		if test "$receivepack"
+		then
+		    git config submodule."$name".receivepack "$receivepack" ||
+		    echo "  Warn: Failed to set receivepack for '$url' in submodule path '$path'."
+		fi
+
 		say "Submodule '$name' ($url) registered for path '$path'"
 	done
 }
@@ -330,7 +390,8 @@ cmd_update()
 
 		if ! test -d "$path"/.git -o -f "$path"/.git
 		then
-			module_clone "$path" "$url" || exit
+			module_clone "$path" "$url" "$(git config submodule."$name".uploadpack)" \
+			    "$(git config submodule."$name".receivepack)" || exit
 			subsha1=
 		else
 			subsha1=$(unset GIT_DIR; cd "$path" &&
@@ -655,6 +716,18 @@ cmd_sync()
 			remote=$(get_default_remote)
 			say "Synchronizing submodule url for '$name'"
 			git config remote."$remote".url "$url"
+			uploadpack=$(git config -f .gitmodules submodule."$name".uploadpack)
+			receivepack=$(git config -f .gitmodules submodule."$name".receivepack)
+			if test "$uploadpack"
+			then
+			    git config submodule."$name".uploadpack "$uploadpack" ||
+			    echo "  Warn: Failed to set uploadpack for '$url' in submodule path '$name'."
+			fi
+			if test "$receivepack"
+			then
+			    git config submodule."$name".receivepack "$receivepack" ||
+			    echo "  Warn: Failed to set receivepack for '$url' in submodule path '$name'."
+			fi
 		)
 		fi
 	done
diff --git a/t/t7404-submodule-packbin.sh b/t/t7404-submodule-packbin.sh
new file mode 100755
index 0000000..d46b3e6
--- /dev/null
+++ b/t/t7404-submodule-packbin.sh
@@ -0,0 +1,53 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 E. Jason Riedy
+#
+
+test_description='git submodule with explicit pack programs
+
+These tests exercise git submodule with --upload-pack and --receive-pack arguments.
+'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	echo file > file &&
+	git add file &&
+	test_tick &&
+	git commit -m upstream
+	git clone . super &&
+	git clone super submodule &&
+	(cd super &&
+	 git submodule add --upload-pack "${GIT_EXEC_PATH}/git-upload-pack" --receive-pack "${GIT_EXEC_PATH}/git-receive-pack" ../submodule submodule &&
+	 test_tick &&
+	 git commit -m "submodule"
+	) &&
+	git clone super super-clone &&
+	(cd super-clone && git submodule update --init)
+'
+
+test_expect_success 'push submodule change' '
+	(cd super &&
+         cd submodule &&
+         git checkout master &&
+	 echo second line >> file &&
+	 test_tick &&
+	 git commit -a -m "change submodule inside" &&
+         git push origin +master:pushed
+	)
+'
+
+test_expect_success 'pull submodule change' '
+	(cd submodule &&
+         git pull . pushed &&
+	 echo second line >> file &&
+	 test_tick &&
+	 git commit -a -m "change submodule outside"
+	) &&
+        (cd super &&
+         cd submodule &&
+         git pull origin master
+        )
+'
+
+test_done
-- 
1.6.0.2

[PATCH] Pass --upload-pack and --receive-pack through submodules.

[Jason Riedy]

While I no longer have to worry about a zillion ancient OS versions,
I now have to worry about a remote site where I cannot control the
path for non-interactive shells.  Thus, submodules need to handle
explicitly specified git-upload-pack and git-receive-pack programs.

There may be a fun shell quoting dance to avoid the extra conditionals.
Any such dance would be even more confusing to read.

Signed-off-by: Jason Riedy <jason@acm.org>
---
  Sorry; I forgot the documentation patch last time.  And it might be
  better just to copy the entire submodule config directly to the remote
  section...

 Documentation/git-submodule.txt |   15 +++++++-
 Documentation/gitmodules.txt    |   13 ++++++-
 git-submodule.sh                |   81 +++++++++++++++++++++++++++++++++++++--
 t/t7404-submodule-packbin.sh    |   53 +++++++++++++++++++++++++
 4 files changed, 156 insertions(+), 6 deletions(-)
 create mode 100755 t/t7404-submodule-packbin.sh

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 2f207fb..9e2de16 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -9,7 +9,7 @@ git-submodule - Initialize, update or inspect submodules
 SYNOPSIS
 --------
 [verse]
-'git submodule' [--quiet] add [-b branch] [--] <repository> <path>
+'git submodule' [--quiet] add [-b branch] [-u <git-upload-pack>] [--receive-pack <git-receive-pack>] [--] <repository> <path>
 'git submodule' [--quiet] status [--cached] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
 'git submodule' [--quiet] update [--init] [--] [<path>...]
@@ -159,6 +159,19 @@ OPTIONS
 --branch::
 	Branch of repository to add as submodule.
 
+--upload-pack <upload-pack>::
+-u <upload-pack>::
+	When given, and the repository to clone from is accessed
+	via ssh, this specifies a non-default path for the
+	'git-upload-pack' program on the remote end.  See
+	linkgit:git-fetch-pack[1].
+
+--receive-pack <receive-pack>::
+	When given, and the repository to clone from is accessed
+	via ssh, this specifies a non-default path for the
+	'git-receive-pack' program on the remote end.  See
+	linkgit:git-push[1].
+
 --cached::
 	This option is only valid for status and summary commands.  These
 	commands typically use the commit found in the submodule HEAD, but
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index d1a17e2..3387951 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -30,6 +30,13 @@ submodule.<name>.path::
 submodule.<name>.url::
 	Defines an url from where the submodule repository can be cloned.
 
+submodule.<name>.receivepack::
+	The default program to execute on the remote side when pushing.  See
+	option \--receive-pack of linkgit:git-push[1].
+
+submodule.<name>.uploadpack::
+	The default program to execute on the remote side when fetching.  See
+	option \--upload-pack of linkgit:git-fetch-pack[1].
 
 EXAMPLES
 --------
@@ -42,12 +49,16 @@ Consider the following .gitmodules file:
 
 	[submodule "libbar"]
 		path = include/bar
-		url = git://bar.com/git/lib.git
+		url = ssh://bar.com/~/git/lib.git
+		uploadpack = /home/you/bin/git-upload-pack-wrapper
+		receivepack = /home/you/bin/git-receive-pack-wrapper
 
 
 This defines two submodules, `libfoo` and `libbar`. These are expected to
 be checked out in the paths 'include/foo' and 'include/bar', and for both
 submodules an url is specified which can be used for cloning the submodules.
+For `libbar`, packs are retrieved and stored via the upload and receive
+wrappers, respectively.
 
 SEE ALSO
 --------
diff --git a/git-submodule.sh b/git-submodule.sh
index 2f47e06..1a8a968 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -90,6 +90,8 @@ module_clone()
 {
 	path=$1
 	url=$2
+	uploadpack=$3
+	receivepack=$4
 
 	# If there already is a directory at the submodule path,
 	# expect it to be empty (since that is the default checkout
@@ -105,8 +107,23 @@ module_clone()
 	test -e "$path" &&
 	die "A file already exist at path '$path'"
 
-	git-clone -n "$url" "$path" ||
+	if test "$uploadpack"
+	then
+	    git-clone --upload-pack $uploadpack -n "$url" "$path"
+	else
+	    git-clone -n "$url" "$path"
+	fi ||
 	die "Clone of '$url' into submodule path '$path' failed"
+	if test "$uploadpack"
+	then
+	    git config -f "${path}/.git/config" remote.origin.uploadpack "$uploadpack" ||
+	    echo "  Warn: Failed to set uploadpack for '$url' in submodule path '$path'."
+	fi
+	if test "$receivepack"
+	then
+	    git config -f "${path}/.git/config" remote.origin.receivepack "$receivepack" ||
+	    echo "  Warn: Failed to set receivepack for '$url' in submodule path '$path'."
+	fi
 }
 
 #
@@ -130,6 +147,16 @@ cmd_add()
 		-q|--quiet)
 			quiet=1
 			;;
+		-u|--upload-pack)
+			case "$2" in '') usage ;; esac
+			uploadpack=$2
+			shift
+			;;
+		--receive-pack)
+			case "$2" in '') usage ;; esac
+			receivepack=$2
+			shift
+			;;
 		--)
 			shift
 			break
@@ -191,9 +218,17 @@ cmd_add()
 			;;
 		esac
 		git config submodule."$path".url "$url"
+		if test "$uploadpack"
+		then
+		    git config submodule."$path".uploadpack "$uploadpack"
+		fi
+		if test "$receivepack"
+		then
+		    git config submodule."$path".receivepack "$receivepack"
+		fi
 	else
 
-		module_clone "$path" "$realrepo" || exit
+		module_clone "$path" "$realrepo" "$uploadpack" "$receivepack" || exit
 		(unset GIT_DIR; cd "$path" && git checkout -f -q ${branch:+-b "$branch" "origin/$branch"}) ||
 		die "Unable to checkout submodule '$path'"
 	fi
@@ -202,7 +237,19 @@ cmd_add()
 	die "Failed to add submodule '$path'"
 
 	git config -f .gitmodules submodule."$path".path "$path" &&
-	git config -f .gitmodules submodule."$path".url "$repo" &&
+	git config -f .gitmodules submodule."$path".url "$repo" ||
+	die "Failed to register submodule '$path'"
+
+	if test "$uploadpack"
+        then
+	    git config -f .gitmodules submodule."$path".uploadpack "$uploadpack" ||
+	    die "Failed to register submodule '$path'"
+	fi
+	if test "$receivepack"
+        then
+	    git config -f .gitmodules submodule."$path".receivepack "$receivepack" ||
+	    die "Failed to register submodule '$path'"
+	fi
 	git add .gitmodules ||
 	die "Failed to register submodule '$path'"
 }
@@ -277,6 +324,19 @@ cmd_init()
 		git config submodule."$name".url "$url" ||
 		die "Failed to register url for submodule path '$path'"
 
+		uploadpack=$(git config -f .gitmodules submodule."$name".uploadpack)
+		receivepack=$(git config -f .gitmodules submodule."$name".receivepack)
+		if test "$uploadpack"
+		then
+		    git config submodule."$name".uploadpack "$uploadpack" ||
+		    echo "  Warn: Failed to set uploadpack for '$url' in submodule path '$path'."
+		fi
+		if test "$receivepack"
+		then
+		    git config submodule."$name".receivepack "$receivepack" ||
+		    echo "  Warn: Failed to set receivepack for '$url' in submodule path '$path'."
+		fi
+
 		say "Submodule '$name' ($url) registered for path '$path'"
 	done
 }
@@ -330,7 +390,8 @@ cmd_update()
 
 		if ! test -d "$path"/.git -o -f "$path"/.git
 		then
-			module_clone "$path" "$url" || exit
+			module_clone "$path" "$url" "$(git config submodule."$name".uploadpack)" \
+			    "$(git config submodule."$name".receivepack)" || exit
 			subsha1=
 		else
 			subsha1=$(unset GIT_DIR; cd "$path" &&
@@ -655,6 +716,18 @@ cmd_sync()
 			remote=$(get_default_remote)
 			say "Synchronizing submodule url for '$name'"
 			git config remote."$remote".url "$url"
+			uploadpack=$(git config -f .gitmodules submodule."$name".uploadpack)
+			receivepack=$(git config -f .gitmodules submodule."$name".receivepack)
+			if test "$uploadpack"
+			then
+			    git config submodule."$name".uploadpack "$uploadpack" ||
+			    echo "  Warn: Failed to set uploadpack for '$url' in submodule path '$name'."
+			fi
+			if test "$receivepack"
+			then
+			    git config submodule."$name".receivepack "$receivepack" ||
+			    echo "  Warn: Failed to set receivepack for '$url' in submodule path '$name'."
+			fi
 		)
 		fi
 	done
diff --git a/t/t7404-submodule-packbin.sh b/t/t7404-submodule-packbin.sh
new file mode 100755
index 0000000..d46b3e6
--- /dev/null
+++ b/t/t7404-submodule-packbin.sh
@@ -0,0 +1,53 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 E. Jason Riedy
+#
+
+test_description='git submodule with explicit pack programs
+
+These tests exercise git submodule with --upload-pack and --receive-pack arguments.
+'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	echo file > file &&
+	git add file &&
+	test_tick &&
+	git commit -m upstream
+	git clone . super &&
+	git clone super submodule &&
+	(cd super &&
+	 git submodule add --upload-pack "${GIT_EXEC_PATH}/git-upload-pack" --receive-pack "${GIT_EXEC_PATH}/git-receive-pack" ../submodule submodule &&
+	 test_tick &&
+	 git commit -m "submodule"
+	) &&
+	git clone super super-clone &&
+	(cd super-clone && git submodule update --init)
+'
+
+test_expect_success 'push submodule change' '
+	(cd super &&
+         cd submodule &&
+         git checkout master &&
+	 echo second line >> file &&
+	 test_tick &&
+	 git commit -a -m "change submodule inside" &&
+         git push origin +master:pushed
+	)
+'
+
+test_expect_success 'pull submodule change' '
+	(cd submodule &&
+         git pull . pushed &&
+	 echo second line >> file &&
+	 test_tick &&
+	 git commit -a -m "change submodule outside"
+	) &&
+        (cd super &&
+         cd submodule &&
+         git pull origin master
+        )
+'
+
+test_done
-- 
1.6.1.60.g1f086.dirty

Re: [PATCH] Pass --upload-pack and --receive-pack through submodules.

[Junio C Hamano]

Jason Riedy <jason@acm.org> writes:

> While I no longer have to worry about a zillion ancient OS versions,
> I now have to worry about a remote site where I cannot control the
> path for non-interactive shells.  Thus, submodules need to handle
> explicitly specified git-upload-pack and git-receive-pack programs.

Do we (and can we) assume that the remote repositories submodules fetch
from all reside on the same host and can share the same values for these
parameters?  Shouldn't these instead be specified in the configuration
files for the submodules, if they need to be nonstandard values?

Re: [PATCH] Pass --upload-pack and --receive-pack through submodules.

[Jason Riedy]

And Junio C. Hamano writes:
> Do we (and can we) assume that the remote repositories submodules fetch
> from all reside on the same host and can share the same values for these
> parameters?  Shouldn't these instead be specified in the configuration
> files for the submodules, if they need to be nonstandard values?

git does not and (for me) should not make the assumption above.
I'm fetching different pieces from different administrative
domains where I also have push access.  While I could clone them
all into one place, that adds another level of repositories I'd
have to keep up-to-date.

Essentially, I'm collecting experimental libraries and drivers
into one build tree.  My intent is to record exactly what is
being built on multiple machines while being able to push local
build configurations back to my central superproject.  Then I can
make a version with pull-only remote URLs[1] for others to
duplicate the setup for my experiments.

I suppose I could clone the submodules by hand and then use git
submodule add.  But then I'll have to build the structure by hand
(with my own wrapper) everywhere I use the supermodule.  At that
point, using git submodule becomes a tad silly.  Or I could set
up the superproject with pull-only URLs and modify the
configurations with a separate wrapper.  Seemed like adding the
two options to git submodule was easier.

Jason

Footnotes: 
[1]  Or ask for separate pull- and push-urls in git remote.  But
I haven't thought that through.