[PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Mark Levedahl]

(sigh, someday I'll get this command line email addressing stuff
sorted out - resending as I mangled Junio's email)

This patch series was inspired by several issues dealing with
multiple remotes and submodules. The tests added as the last
patch demonstrate the specific use pattern to be supported,
and that does not work currently.

1 - user A creates a project on server "frotz" with a submodule.
2 - user B clones the project, forks it, adds a submodule, and
publishes the fork on server "fork"

user A adds the fork to his working tree, via git remote add +
git fetch. The desire is that:
1 - git submodule init + git submodule update include the new
submdule added in the fork.
2 - The top-level project and all submodules use the same
remote nickname to refer to the same server.

Breakage in 1.5.4 without this patch series is in several places:

1 - git submodule does not follow the top-level project's
branch.<name>.remote in fetching, so does not try to clone
the new submodule from remoe "fork" where the fork exists.
2 - If 1 were fixed, git submodule still invokes clone without "-o
<remotenick>" when cloning, so each submodule would have "origin"
point to a different remote.
3 - If 2 were fixed, submodule update leaves submodules on detached
heads, so the remotes machinery cannot supply a remote name, and thus
"origin" is used for further updates. As origin may not be defined,
(e.g., for the submodule fetched from "fork" this also fails.)

The underlying design issue is the assumption that all projects
revolve around a single server, so that *the* server can be called
"origin" without confusion.  Given a project with multiple servers and
no one server being *the* server, origin cannot be usefully defined.
Absent submodules, the remotes machinery largely masks this: origin is
referred to only as the default remote and it is easy to avoid using
the default remote.  With submodules, this becomes more of an issue.

Following this patch series, the following is true:

1) The existing workflow of top-level and all sub-modules always
cloned from and interacting with a single default remote
nick-named "origin" is unchanged.
2) Via clone -o <nick> (or git confit core.origin <nick>), the
default remote is changed from origin to <nick>, and that default
is used instead of origin everywhere.
3) git-submodule uses the top-level's branch.<name>.remote to
choose which remote to fetch from, uses that in all submodules,
including defining that in existing submodules whose url is
relative (../* or ./*) indicating the submodule is maintained on
same server as top-level.

The penultimate patch addresses a basic usability issue, allowing
addition of a submodule in place in a project, rather than requiring
it be cloned from a remote.  This hole in the porcelain became evident
in creating a test (t7401) for this series.
submodules except by cloning).

[PATCH 1/7] Teach remote machinery about core.origin config variable

[Mark Levedahl]

This introduces a new configuration variable, core.origin, that
defines the name of the default remote to be used. Traditionally, this
is "origin", and could be overridden for a given branch. This change
introduces a way to redefine the default as desired and have that honored
regardless of the currently checked out head (e.g., core.origin is
used when on a detached head or any other non-tracking branch).

Signed-off-by: Mark Levedahl <mlevedahl@gmail.com>
---
 Documentation/config.txt |    6 ++++++
 git-parse-remote.sh      |    5 +++--
 remote.c                 |   11 ++++++++++-
 3 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 4e222f1..a890f5d 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -291,6 +291,12 @@ core.editor::
 	`GIT_EDITOR` environment, `core.editor`, `VISUAL` and
 	`EDITOR` environment variables and then finally `vi`.
 
+core.origin::
+	The name of the remote used by default for fetch / pull. If unset,
+	origin is assumed. This value is used whenever the current branch
+	has no corresponding branch.<name>.remote, such as when working on
+	a detached head.
+
 core.pager::
 	The command that git will use to paginate output.  Can be overridden
 	with the `GIT_PAGER` environment variable.
diff --git a/git-parse-remote.sh b/git-parse-remote.sh
index 695a409..c7ac7c7 100755
--- a/git-parse-remote.sh
+++ b/git-parse-remote.sh
@@ -56,8 +56,9 @@ get_remote_url () {
 
 get_default_remote () {
 	curr_branch=$(git symbolic-ref -q HEAD | sed -e 's|^refs/heads/||')
-	origin=$(git config --get "branch.$curr_branch.remote")
-	echo ${origin:-origin}
+	git config --get "branch.$curr_branch.remote" ||
+	git config --get "core.origin" ||
+	echo origin
 }
 
 get_remote_default_refs_for_push () {
diff --git a/remote.c b/remote.c
index 0e00680..302d499 100644
--- a/remote.c
+++ b/remote.c
@@ -10,6 +10,7 @@ static int allocated_branches;
 
 static struct branch *current_branch;
 static const char *default_remote_name;
+static const char *core_origin;
 
 #define BUF_SIZE (2048)
 static char buffer[BUF_SIZE];
@@ -233,6 +234,11 @@ static int handle_config(const char *key, const char *value)
 			add_merge(branch, xstrdup(value));
 		return 0;
 	}
+	if (!strcmp(key, "core.origin")) {
+		if (value)
+			core_origin = xstrdup(value);
+		return 0;
+	}
 	if (prefixcmp(key,  "remote."))
 		return 0;
 	name = key + 7;
@@ -291,7 +297,6 @@ static void read_config(void)
 	int flag;
 	if (default_remote_name) // did this already
 		return;
-	default_remote_name = xstrdup("origin");
 	current_branch = NULL;
 	head_ref = resolve_ref("HEAD", sha1, 0, &flag);
 	if (head_ref && (flag & REF_ISSYMREF) &&
@@ -300,6 +305,10 @@ static void read_config(void)
 			make_branch(head_ref + strlen("refs/heads/"), 0);
 	}
 	git_config(handle_config);
+	if (!default_remote_name) {
+		default_remote_name = core_origin ?
+		core_origin : xstrdup("origin");
+	}
 }
 
 struct refspec *parse_ref_spec(int nr_refspec, const char **refspec)
-- 
1.5.4.18.g43c18

[PATCH 2/7] git-remote - Unset core.origin when deleting the default remote

[Mark Levedahl]

Signed-off-by: Mark Levedahl <mlevedahl@gmail.com>
---
 git-remote.perl |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/git-remote.perl b/git-remote.perl
index d13e4c1..75d2371 100755
--- a/git-remote.perl
+++ b/git-remote.perl
@@ -328,6 +328,11 @@ sub rm_remote {
 
 	$git->command('config', '--remove-section', "remote.$name");
 
+	my $defremote = $git->config("core.origin");
+	if (defined $defremote && $defremote eq $name) {
+	       $git->command("config", "--unset", "core.origin");
+	}
+
 	eval {
 	    my @trackers = $git->command('config', '--get-regexp',
 			'branch.*.remote', $name);
-- 
1.5.4.18.g43c18

[PATCH 3/7] git-clone - Set remotes.origin config variable

[Mark Levedahl]

This records the users choice of default remote name (by default "origin")
as given by the -o option.

Signed-off-by: Mark Levedahl <mlevedahl@gmail.com>
---
 Documentation/git-clone.txt |    3 ++-
 git-clone.sh                |    1 +
 2 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 2341881..f15aecd 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -103,7 +103,8 @@ OPTIONS
 --origin <name>::
 -o <name>::
 	Instead of using the remote name 'origin' to keep track
-	of the upstream repository, use <name> instead.
+	of the upstream repository, use <name> instead. The name
+	is recorded in the core.origin config variable.
 
 --upload-pack <upload-pack>::
 -u <upload-pack>::
diff --git a/git-clone.sh b/git-clone.sh
index b4e858c..7208d68 100755
--- a/git-clone.sh
+++ b/git-clone.sh
@@ -242,6 +242,7 @@ fi &&
 export GIT_DIR &&
 GIT_CONFIG="$GIT_DIR/config" git-init $quiet ${template+"$template"} || usage
 
+git config core.origin $origin
 if test -n "$bare"
 then
 	GIT_CONFIG="$GIT_DIR/config" git config core.bare true
-- 
1.5.4.18.g43c18

[PATCH 4/7] git-submodule - Possibly inherit parent's default remote on init/clone

[Mark Levedahl]

For submodules defined relative to their parent, it is likely that the
parent's defined default remote is correct for the child as well. This
allows use of remote names other than "origin", important as managed
submodules are typically checked out on a detached head and therefore
submodule-update invokes git-fetch using the default remote. Without this
change, submodules effectively had to have a default remote of "origin."

Signed-off-by: Mark Levedahl <mlevedahl@gmail.com>
---
 Documentation/git-submodule.txt |    8 +++++---
 git-submodule.sh                |   20 ++++++++++++++------
 2 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index e818e6e..b04b107 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -37,9 +37,11 @@ status::
 
 init::
 	Initialize the submodules, i.e. register in .git/config each submodule
-	name and url found in .gitmodules. The key used in .git/config is
-	`submodule.$name.url`. This command does not alter existing information
-	in .git/config.
+	name and url found in .gitmodules, along with the default remote origin.
+	For submodules using a relative url, the default remote is inherited
+	from the parent project, for absolute urls the default "origin" is used.
+	The key used in .git/config is submodule.$name.url`. This command does
+	not alter existing information in .git/config.
 
 update::
 	Update the registered submodules, i.e. clone missing submodules and
diff --git a/git-submodule.sh b/git-submodule.sh
index a6aaf40..bb0d265 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -7,6 +7,7 @@
 USAGE='[--quiet] [--cached] [add <repo> [-b branch]|status|init|update] [--] [<path>...]'
 OPTIONS_SPEC=
 . git-sh-setup
+. git-parse-remote
 require_work_tree
 
 command=
@@ -40,9 +41,7 @@ get_repo_base() {
 # Resolve relative url by appending to parent's url
 resolve_relative_url ()
 {
-	branch="$(git symbolic-ref HEAD 2>/dev/null)"
-	remote="$(git config branch.${branch#refs/heads/}.remote)"
-	remote="${remote:-origin}"
+	remote="$(get_default_remote)"
 	remoteurl="$(git config remote.$remote.url)" ||
 		die "remote ($remote) does not have a url in .git/config"
 	url="$1"
@@ -92,6 +91,7 @@ module_clone()
 {
 	path=$1
 	url=$2
+	origin=$3
 
 	# If there already is a directory at the submodule path,
 	# expect it to be empty (since that is the default checkout
@@ -107,7 +107,7 @@ module_clone()
 	test -e "$path" &&
 	die "A file already exist at path '$path'"
 
-	git-clone -n "$url" "$path" ||
+	git-clone -n -o "$origin" "$url" "$path" ||
 	die "Clone of '$url' into submodule path '$path' failed"
 }
 
@@ -156,10 +156,12 @@ cmd_add()
 	case "$repo" in
 	./*|../*)
 		# dereference source url relative to parent's url
+		origin=$(get_default_remote)
 		realrepo="$(resolve_relative_url $repo)" ;;
 	*)
 		# Turn the source into an absolute path if
 		# it is local
+		origin=origin
 		if base=$(get_repo_base "$repo"); then
 			repo="$base"
 		fi
@@ -180,7 +182,7 @@ cmd_add()
 	git ls-files --error-unmatch "$path" > /dev/null 2>&1 &&
 	die "'$path' already exists in the index"
 
-	module_clone "$path" "$realrepo" || exit
+	module_clone "$path" "$realrepo" "$origin" || exit
 	(unset GIT_DIR; cd "$path" && git checkout -q ${branch:+-b "$branch" "origin/$branch"}) ||
 	die "Unable to checkout submodule '$path'"
 	git add "$path" ||
@@ -236,9 +238,14 @@ cmd_init()
 		case "$url" in
 		./*|../*)
 			url="$(resolve_relative_url "$url")"
+			origin=$(get_default_remote)
+			;;
+		*)
+			origin=origin
 			;;
 		esac
 
+		git config submodule."$name".origin "$origin" &&
 		git config submodule."$name".url "$url" ||
 		die "Failed to register url for submodule path '$path'"
 
@@ -287,10 +294,11 @@ cmd_update()
 			say "Submodule path '$path' not initialized"
 			continue
 		fi
+		origin=$(git config submodule."$name".origin || echo origin)
 
 		if ! test -d "$path"/.git
 		then
-			module_clone "$path" "$url" || exit
+			module_clone "$path" "$url" "$origin" || exit
 			subsha1=
 		else
 			subsha1=$(unset GIT_DIR; cd "$path" &&
-- 
1.5.4.18.g43c18

[PATCH 5/7] Teach git-submodule to use top-level remote when updating subprojects

[Mark Levedahl]

Modules that are defined using relative urls to the master project are
assumed to be completely owned by the project. When running
"submodule update" from the top level, it is reasonable that the entire
project exists at the remote given by top-level branch.<name>.remote.
Using the remote machinery, this remote can be different for each
branch and can be different than the current defaults in each submodule.

This teaches submodule to:

1) Possibly define the current master's remote in each submodule, using
the same relative url used by submodule init.
2) Fetch each submodule's updates from the master's remote.

Submodules defined using absolute urls (not relative to the parent) are
not touched by this logic. Such modules are assumed to be independent
of the master project so submodule can do no better than to fetch from
their currently defined default remotes as already done.

Signed-off-by: Mark Levedahl <mlevedahl@gmail.com>
---
 git-submodule.sh |   20 +++++++++++++++++++-
 1 files changed, 19 insertions(+), 1 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index bb0d265..8f31ebe 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -255,6 +255,8 @@ cmd_init()
 
 #
 # Update each submodule path to correct revision, using clone and checkout as needed
+# For owned submodules (defined using relative url), we use master project's remote
+# and define that in each submodule if not already there
 #
 # $@ = requested paths (default to all)
 #
@@ -281,6 +283,7 @@ cmd_update()
 		shift
 	done
 
+	master_remote=$(get_default_remote)
 	git ls-files --stage -- "$@" | grep -e '^160000 ' |
 	while read mode sha1 stage path
 	do
@@ -306,9 +309,24 @@ cmd_update()
 			die "Unable to find current revision in submodule path '$path'"
 		fi
 
+		baseurl="$(GIT_CONFIG=.gitmodules git config submodule."$name".url)"
+		case "$baseurl" in
+		./*|../*)
+			fetch_remote=$master_remote
+			(unset GIT_DIR ; cd "$path" && git config remote."$fetch_remote".url > nul) ||
+			(
+				absurl="$(resolve_relative_url $baseurl)"
+				unset GIT_DIR; cd "$path" && git remote add "$master_remote" "$absurl"
+			) || die "Unable to define remote '$fetch_remote' in submodule path '$path'"
+			;;
+		*)
+			fetch_remote=
+			;;
+		esac
+
 		if test "$subsha1" != "$sha1"
 		then
-			(unset GIT_DIR; cd "$path" && git-fetch &&
+			(unset GIT_DIR; cd "$path" && git-fetch $fetch_remote &&
 				git-checkout -q "$sha1") ||
 			die "Unable to checkout '$sha1' in submodule path '$path'"
 
-- 
1.5.4.18.g43c18

[PATCH 6/7] git-submodule - Allow adding a submodule in-place

[Mark Levedahl]

When working in top-level directory, it is natural to create a new
submodule in a subdirectory, then add that submodule to top-level in
place. This allows "git submodule add <intended url> subdir" to add
the existing subdir to the current project. The presumption is the user
will later push / clone the subdir to the <intended url> so that future
submodule init / updates will work.

Absent this patch, "git submodule add" insists upon cloning the subdir
from the remote, which is fine for adding an existing project in but
less useful when adding a new submodule from scratch to an existing
project. This functionality remains, and the clone is attempted if the
subdir does not already exist as a valid git repo.

Signed-off-by: Mark Levedahl <mlevedahl@gmail.com>

fix submodule again
---
 Documentation/git-submodule.txt |    5 ++-
 git-submodule.sh                |   56 +++++++++++++++++++++++---------------
 2 files changed, 37 insertions(+), 24 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index b04b107..8ea3fc3 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -18,8 +18,9 @@ COMMANDS
 --------
 add::
 	Add the given repository as a submodule at the given path
-	to the changeset to be committed next.  In particular, the
-	repository is cloned at the specified path, added to the
+	to the changeset to be committed next.  If path is a valid
+	repository within the project, it is added as is. Otherwise,
+	repository is cloned at the specified path. path is added to the
 	changeset and registered in .gitmodules.   If no path is
 	specified, the path is deduced from the repository specification.
 	If the repository url begins with ./ or ../, it is stored as
diff --git a/git-submodule.sh b/git-submodule.sh
index 8f31ebe..47b5664 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -44,6 +44,7 @@ resolve_relative_url ()
 	remote="$(get_default_remote)"
 	remoteurl="$(git config remote.$remote.url)" ||
 		die "remote ($remote) does not have a url in .git/config"
+	remoteurl="${remoteurl%/.git}"
 	url="$1"
 	while test -n "$url"
 	do
@@ -153,22 +154,6 @@ cmd_add()
 		usage
 	fi
 
-	case "$repo" in
-	./*|../*)
-		# dereference source url relative to parent's url
-		origin=$(get_default_remote)
-		realrepo="$(resolve_relative_url $repo)" ;;
-	*)
-		# Turn the source into an absolute path if
-		# it is local
-		origin=origin
-		if base=$(get_repo_base "$repo"); then
-			repo="$base"
-		fi
-		realrepo=$repo
-		;;
-	esac
-
 	# Guess path from repo if not specified or strip trailing slashes
 	if test -z "$path"; then
 		path=$(echo "$repo" | sed -e 's|/*$||' -e 's|:*/*\.git$||' -e 's|.*[/:]||g')
@@ -176,15 +161,42 @@ cmd_add()
 		path=$(echo "$path" | sed -e 's|/*$||')
 	fi
 
-	test -e "$path" &&
-	die "'$path' already exists"
-
 	git ls-files --error-unmatch "$path" > /dev/null 2>&1 &&
 	die "'$path' already exists in the index"
 
-	module_clone "$path" "$realrepo" "$origin" || exit
-	(unset GIT_DIR; cd "$path" && git checkout -q ${branch:+-b "$branch" "origin/$branch"}) ||
-	die "Unable to checkout submodule '$path'"
+	# perhaps the path exists and is already a git repo, else clone it
+	if test -e "$path"
+	then
+		if test -d "$path/.git" &&
+		test "$(unset GIT_DIR; cd $path; git rev-parse --git-dir)" = ".git"
+		then
+			echo "Adding existing repo at '$path' to the index"
+		else
+			die "'$path' already exists and is not a valid git repo"
+		fi
+	else
+		case "$repo" in
+		./*|../*)
+			# dereference source url relative to parent's url
+			origin=$(get_default_remote)
+			realrepo="$(resolve_relative_url $repo)" ;;
+		*)
+			# Turn the source into an absolute path if
+			# it is local
+			origin=origin
+			if base=$(get_repo_base "$repo")
+			then
+				repo="$base"
+			fi
+			realrepo=$repo
+			;;
+		esac
+
+		module_clone "$path" "$realrepo" "$origin" || exit
+		(unset GIT_DIR; cd "$path" && git checkout -q ${branch:+-b "$branch" "origin/$branch"}) ||
+		die "Unable to checkout submodule '$path'"
+	fi
+
 	git add "$path" ||
 	die "Failed to add submodule '$path'"
 
-- 
1.5.4.18.g43c18

[PATCH 7/7] Add t/t7401 - test submodule interaction with remotes machinery

[Mark Levedahl]

This adds a sequence of tests to assure that the following two sequences
work:

	git clone -o frotz <someurl> foo
	cd foo
	git submodule init
	git submodule update

This should result in the master and subproject having "frotz" as the
name of the default remote (and origin undefined).

Then, in the same working directory
	git remote add fork <some url>
	git fetch fork
	git checkout --track -b fork fork/<somebranch>
	git submodule init
	git submodule update

will retrive new submodules from remote "fork", and define fork in the
existing modules. Origin remains undefined.

Note: this latter case is a clear motivation for overriding "origin": after
the second test, the various submodules would have different ideas of
remote "origin": these would point to different servers. This would
entirely prevent the top-level branch.<name>.remote machinery from
controlling the project as there is no uniform naming of remotes.

Signed-off-by: Mark Levedahl <mlevedahl@gmail.com>
---
 t/t7401-submodule-remote.sh |  104 +++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 104 insertions(+), 0 deletions(-)
 create mode 100755 t/t7401-submodule-remote.sh

diff --git a/t/t7401-submodule-remote.sh b/t/t7401-submodule-remote.sh
new file mode 100755
index 0000000..1a793db
--- /dev/null
+++ b/t/t7401-submodule-remote.sh
@@ -0,0 +1,104 @@
+#!/bin/sh
+
+test_description='Porcelain support for submodules with multiple remotes
+
+This test verifies operation of submodules using multiple remotes and
+differing remote per top-level branch.  This includes ability to name the
+default something other than origin, to follow the top-level
+remote.<branch>, and to propagate definition of new remotes down to
+submodules as needed.
+'
+
+. ./test-lib.sh
+
+# the standard tests all work with one repo, but we need several..
+rm -rf .git
+
+test_expect_success 'Prepare master repository with 1 submodule' '
+	(
+	mkdir master &&
+	cd master &&
+	git init &&
+	echo "on master" > master.txt &&
+	git add master.txt &&
+	git commit -m "Add master.txt" &&
+	mkdir submod1 &&
+	cd submod1 &&
+	git init &&
+	echo "submod1" > submod1.txt &&
+	git add submod1.txt &&
+	git commit -m "Added submod1.txt" &&
+	cd .. &&
+	git submodule add ./submod1 submod1 &&
+	git commit -m "Added submodule submod1"
+	)
+'
+
+test_expect_success 'Clone master as fork' '
+	(
+	git clone master fork &&
+	cd fork &&
+	test "$(git remote)" = "origin" &&
+	git submodule init &&
+	git submodule update &&
+	test -e submod1/.git
+	)
+'
+
+test_expect_success 'Add second submodule in fork' '
+	(
+	cd fork &&
+	mkdir submod2 &&
+	cd submod2 &&
+	git init &&
+	echo "submod2" > submod2.txt &&
+	git add submod2.txt &&
+	git commit -m "Added submod2.txt" &&
+	cd .. &&
+	git submodule add ./submod2 submod2 &&
+	git commit -m "Added submodule submod2 on fork"
+	)
+'
+
+test_expect_success 'Clone master using frotz instead of origin' '
+	(
+	git clone -o frotz master worker &&
+	cd worker &&
+	test "$(git remote)" = "frotz"
+	)
+'
+
+test_expect_success 'Get submodules using frotz instead of origin' '
+	(
+	cd worker &&
+	git submodule init &&
+	git submodule update &&
+	test -e submod1/.git &&
+	cd submod1 &&
+	test "$(git remote)" = "frotz"
+	)
+'
+
+test_expect_success 'Update using fork to get additional submodule' '
+	(
+	cd worker &&
+	git remote add fork $(pwd)/../fork &&
+	git fetch fork &&
+	git checkout --track -b fork_master fork/master &&
+	git submodule init &&
+	git submodule update &&
+	test -e submod2/.git &&
+	cd submod2 &&
+	test "$(git remote)" = "fork" &&
+	cd ../submod1 &&
+	remotes1=$(git remote) &&
+	case $remotes1 in
+		fork*frotz|frotz*fork)
+			true ;;
+		*)
+			false ;;
+	esac
+	)
+'
+
+test_done
-- 
1.5.4.18.g43c18

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Johannes Schindelin]

Hi,

On Sun, 3 Feb 2008, Mark Levedahl wrote:

> This patch series was inspired by several issues dealing with multiple 
> remotes and submodules. The tests added as the last patch demonstrate 
> the specific use pattern to be supported, and that does not work 
> currently.

It seems that everytime you send a new patch series, it gets longer/more 
complicated/affects more and more of git.

As far as I understood, your problem was _purely_ a submodule issue.  I 
find it utterly unnerving that you keep trying to sneak in _anything_ 
unrelated to submodules.

Such as the origin issue.

I am getting pretty angry that you keep trying to complicating things in 
that area!  origin is origin is origin.  It is the default remote.  It is 
what you can change to fetch from somewhere else _by default_.  So I 
absolutely detest your changes in that regard, who help _noboy_ except 
people who like confusion.

So just change remote.origin.url, but do not mess up our nice source code.

If your issue is with git-submodule, then fix _that_, and do not "fix" 
_anything_ else.

Thankyouverymuch,
Dscho

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Mark Levedahl]

Johannes Schindelin wrote:
> Hi,
>
> As far as I understood, your problem was _purely_ a submodule issue.  I 
> find it utterly unnerving that you keep trying to sneak in _anything_ 
> unrelated to submodules.
>   
It is not purely about submodules, but in fact about a centralized 
concept (named "origin") disrupting a distributed workflow using 
submodules in a distributed version control system.
> I am getting pretty angry that you keep trying to complicating things in 
> that area!  
It is easy to substitute emotion for sound technical arguments. Please 
don't, it helps no one.

Mark

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Johannes Schindelin]

Hi,

On Sun, 3 Feb 2008, Mark Levedahl wrote:

> Johannes Schindelin wrote:
> 
> > As far as I understood, your problem was _purely_ a submodule issue.  
> > I find it utterly unnerving that you keep trying to sneak in 
> > _anything_ unrelated to submodules.
>
> It is not purely about submodules, but in fact about a centralized 
> concept (named "origin") disrupting a distributed workflow using 
> submodules in a distributed version control system.

The "origin" concept has nothing to do with a centralised concept.  It is 
there _purely_ for convenience.  If you want to fetch/pull from somewhere, 
you _have_ to specify from where, _except_ if you use the default.

So your argument here is absolutely bogus.

> > I am getting pretty angry that you keep trying to complicating things 
> > in that area!
>
> It is easy to substitute emotion for sound technical arguments. Please 
> don't, it helps no one.

I take it very personal if you ignore me and my arguments, so please 
don't.  My apologies for answering emotionally.  (I could even say that it 
is easier to ignore sound technical arguments than to substitute emotion 
for them.)

My recommendation: fix submodule, and submodule only.  For your specific 
needs.  Hint: you do not need to introduce core.origin for that (as 
should have become totally obvious by now).

Should something like core.origin still be an issue (which I doubt, for 
the same reason you do not change the _name_ of the environment variable 
$HOME), we can still continue discussing that.

Hth,
Dscho

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Mark Levedahl]

On Feb 4, 2008 9:48 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
>
> Should something like core.origin still be an issue (which I doubt, for
> the same reason you do not change the _name_ of the environment variable
> $HOME), we can still continue discussing that.
>
> Hth,
> Dscho
>
>
BIG difference here:
$HOME = <name> of home directory, <name> is arbitrary. Files there are
found as $HOME/<foo> or <name>/foo, they are the same, and I am free
to set HOME=<arbitrary name>.

Currently
$ORIGIN=origin, i.e., the default is *named* origin, not the default
name is given by $ORIGIN.

Why does this matter? git stores the remote's branches under
refs/remotes/<name>/*. If name=$ORIGIN, I am happy, but the problem is
$ORIGIN=origin, and you argue (strongly) that it MUST be origin. Thus,
branches from server "frotz" *might* be in refs/remotes/frotz/*, or
they *might* be in refs/remotes/origin/*, and that is the root of the
problem.

By the same argument, you have hardcoded in your shell that $HOME=/home, right?

Mark

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Johannes Schindelin]

Hi,

On Mon, 4 Feb 2008, Mark Levedahl wrote:

> On Feb 4, 2008 9:48 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>
> > Should something like core.origin still be an issue (which I doubt, 
> > for the same reason you do not change the _name_ of the environment 
> > variable $HOME), we can still continue discussing that.

Actually, I said more things.  Things about fixing git-submodule first, so 
that you can ask it to update via different remotes than "origin".  And 
then try to argue for the rest.

> BIG difference here:
> $HOME = <name> of home directory, <name> is arbitrary. Files there are
> found as $HOME/<foo> or <name>/foo, they are the same, and I am free
> to set HOME=<arbitrary name>.
> 
> Currently
> $ORIGIN=origin, i.e., the default is *named* origin, not the default
> name is given by $ORIGIN.

"origin" is just the name of the default origin, holding the remote URL.  
Just like "HOME" is just the name of the variable holding the home 
directory.

But really.  Fix git-submodule first.

Ciao,
Dscho

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Junio C Hamano]

"Mark Levedahl" <mlevedahl@gmail.com> writes:

> BIG difference here:
> $HOME = <name> of home directory, <name> is arbitrary. Files there are
> found as $HOME/<foo> or <name>/foo, they are the same, and I am free
> to set HOME=<arbitrary name>.

I think you misunderstood Dscho's $HOME analogy.

For example, the value of HOME environment variable is used as a
shorthand to where you go to when you say

	$ cd

Everybody is happy that the variable is called HOME, and nobody
would complain that the shell does not have another mechanism to
let you have this in your .bashrc

	#!/bin/bash
	HOMEENVVAR=BASE
	BASE=/home/mark
        export HOMEENVVAR BASE

and make the parameterless "cd" honor the environment variable
named with HOMEENVVER (in this case, BASE) instead.

Remote shorthand is exactly like a variable name.  We say "git
pull origin" and do not say "git pull $origin", but that is just
a syntax issue.  And "git pull" and friends are defined to
default to the value of the origin 'variable' exactly like "cd"
is defined to default to the value of the HOME variable.

I would agree 100% with Dscho on this point, if we did not have
"clone -o".

"clone -o" is like the above HOMEENVVAR variable.  It lets you
rename what name other than origin is used to name what the
origin 'variable' usually names, or at least pretends to let you
do so.  But its effect may not extend far enough and if there is
such inconsistency, that is a bug worth fixing.  For example, I
think "git fetch" is Ok for basic use, because "clone -o" writes
branch.master.remote in .git/config that points to the name you
gave instead of 'origin', but there may be places that do use
hardwired 'origin' and nothing else.

And that is the only reason I am still responding to this
thread.

If a supermodule wants a specific remote to be used in a
submodule, it should not default to 'origin' but it should name
that specific remote explicitly.  Even if we added the mechanism
that looks like HOMEENVVAR to tell commands other than "git
fetch" to default to something other than 'origin', I do not
think such a use of submodule should depend on that fallback.  I
still haven't dismissed Dscho's argument that submodule related
issues should be handled inside git-submodule for that reason.

We could remove support for "clone -o" and consistently use
'origin' as the default everywhere and completely remove the
half-baked HOMEENVVAR lookalike.  I do not think such a move is
a good idea, but the point is that the solution to your
submodule problem should work even if we did so.  If the
supermodule and a set of submodules are configured to use a
specific remote that is not 'origin', it should use that
specific remote by _naming_ it, not relying on where the
fallback value is taken by other unrelated parts of the system,
no?

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Johannes Schindelin]

Hi,

On Mon, 4 Feb 2008, Junio C Hamano wrote:

> I would agree 100% with Dscho on this point, if we did not have "clone 
> -o".

AFAICT "clone -o" was meant _only_ for this scenario:

You have a slow connection, but someone can provide you with a thumb 
drive, having the current repository.

You clone it from the thumb drive, but of course you want to use your 
(slow) connection for subsequent updates (i.e. you want to have a 
different default remote).

What you do is

	$ git clone -o thumbdrive /media/usbdisk/that-repository.git/

Then, to be able to update from your preferred remote quickly, you add the 
proper remote:

	$ git remote add --fetch origin git://git.kernel.org/shlub.git/

In my understanding, this scenario is the only reason to live for clone's 
-o option, and it would be actively wrong for it to change the _name_ of 
the default remote to "thumbdrive".

Now, Steffen understood that Mark tries to avoid the use of "origin".  
But why, then, does he wish for _the same_ kettle, but call it chair?

I mean, I can understand if you want to update your submodules from 
another server.  What I do then, is to go into that subdirectory, fetch 
from that other server, and that's it.

I can even understand that you have to update your submodules sometimes 
from here, sometimes from there.

Fine.  You just add remotes for the different servers, cd into the 
submodules, and fetch from them.

I could even see (but not use it myself) that you want to fetch a set of 
submodules from server A on Mondays, from server B on Tuesdays, and from 
server C on Wednesdays, and want to have a nice shortcut for that.  Of 
course, I'd use aliases.  But I can see that people want a "--remote" 
option to "git submodule update".

However, what I cannot see _at all_ is that you have a default remote in a 
set (or all) submodules, and wish to change a perfectly fine terminology 
[*1*].

I cannot see why you want to have a default remote to fetch from, and not 
call it "origin", which would keep things easy and simple.  "What is the 
remote called that 'git fetch' fetches from, by default?" -- "origin".

Simple.  Nice.  Consistent.

My understanding is that a patch is needed for _git-submodule_, to support 
that workflow (the one with servers A, B and C), even if I would not use 
that option.

What we do not need is to confuse git users, what with all those 
complaints that git is too complicated.

Hopefully I finally made myself clear.

Ciao,
Dscho

*1*: I heard the argument put forth that giving the default remote a 
different name, so that people know what server they pull from.  That's 
just, uh, unbelievable.  If it is the default remote the user is fetching 
from, all the user will call is "git fetch", and not even _know_ the 
nickname.

Besides, if you call it "berlin", the _url_ can still point to 
git://git.beijing.cn/great-wall.git/.

Another besides: if you ask the user to look up the nickname of the 
default remote, you can as well ask the user to look up the url of 
"origin".

There is _no need_ to complicate git for a workflow that could be solved 
in a more elegant and less error-prone way to begin with.

P.S.: sorry for this long mail.  I usually like to make a concise point.  
But in this discussion, it got me misunderstood, ignored, or both.

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> AFAICT "clone -o" was meant _only_ for this scenario:
>
> You have a slow connection, but someone can provide you with a thumb 
> drive, having the current repository.
>
> You clone it from the thumb drive, but of course you want to use your 
> (slow) connection for subsequent updates (i.e. you want to have a 
> different default remote).
>
> What you do is
>
> 	$ git clone -o thumbdrive /media/usbdisk/that-repository.git/
>
> Then, to be able to update from your preferred remote quickly, you add the 
> proper remote:
>
> 	$ git remote add --fetch origin git://git.kernel.org/shlub.git/
>
> In my understanding, this scenario is the only reason to live for clone's 
> -o option, and it would be actively wrong for it to change the _name_ of 
> the default remote to "thumbdrive".

Then we should fix "git-clone".

If you do the first command, I am reasonably sure that you would
get:

	[branch "master"]
        	remote = thumbdrive

in your .git/config.  You may want to occasionally go back to
thumbdrive for sneakernetting, so I do not think we should
remove [remote "thumbdrive"] section the clone would leave, but
the above "master branch would interact with thumbdrive
repository" should not be there.

And that corrected behaviour and the intended use of -o should
be documented.

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Johannes Schindelin]

Hi,

On Mon, 4 Feb 2008, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > AFAICT "clone -o" was meant _only_ for this scenario:
> >
> > You have a slow connection, but someone can provide you with a thumb 
> > drive, having the current repository.
> >
> > You clone it from the thumb drive, but of course you want to use your 
> > (slow) connection for subsequent updates (i.e. you want to have a 
> > different default remote).
> >
> > What you do is
> >
> > 	$ git clone -o thumbdrive /media/usbdisk/that-repository.git/
> >
> > Then, to be able to update from your preferred remote quickly, you add 
> > the proper remote:
> >
> > 	$ git remote add --fetch origin git://git.kernel.org/shlub.git/
> >
> > In my understanding, this scenario is the only reason to live for 
> > clone's -o option, and it would be actively wrong for it to change the 
> > _name_ of the default remote to "thumbdrive".
> 
> Then we should fix "git-clone".
> 
> If you do the first command, I am reasonably sure that you would get:
> 
> 	[branch "master"]
>         	remote = thumbdrive
> 
> in your .git/config.  You may want to occasionally go back to thumbdrive 
> for sneakernetting, so I do not think we should remove [remote 
> "thumbdrive"] section the clone would leave, but the above "master 
> branch would interact with thumbdrive repository" should not be there.
> 
> And that corrected behaviour and the intended use of -o should be 
> documented.

Fully agree.  It is now point 38 on my ever-growing TODO list.

Ciao,
Dscho

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Steffen Prohaska]

On Feb 3, 2008, at 11:43 PM, Johannes Schindelin wrote:

> On Sun, 3 Feb 2008, Mark Levedahl wrote:
>
>> This patch series was inspired by several issues dealing with  
>> multiple
>> remotes and submodules. The tests added as the last patch demonstrate
>> the specific use pattern to be supported, and that does not work
>> currently.
>
> It seems that everytime you send a new patch series, it gets longer/ 
> more
> complicated/affects more and more of git.
>
> As far as I understood, your problem was _purely_ a submodule  
> issue.  I
> find it utterly unnerving that you keep trying to sneak in _anything_
> unrelated to submodules.
>
> Such as the origin issue.

As fas as I understood, Mark's usage of submodules might have
revealed a deficiency in the current handling of origin.

Mark described a distributed workflow that actively uses more than
a single server to store repositories and these different servers
are not pure replicates of each other.  As a consequence the name
of the server has a meaning for the users and therefore the workflow
makes this explicit by never using "origin" but instead use
descriptive names for the remotes.  He described situations when
people want to speak about what they have in their repositories
and want to describe where they pulled from.  I can well imaging
that the result of everyone saying "I pulled from origin" can cause
great confusion if origin means different things; and therefore it
is a reasonable choice to completely avoid origin.

Partly because of this explanation by Mark and the problems he
described, I decided against using submodules, but instead merged
all the submodules that I originally planned to keep separate into
a single monolithic repository.  I expect that a single repository
works well because this is what the Linux kernel uses.


> I am getting pretty angry that you keep trying to complicating  
> things in
> that area!  origin is origin is origin.  It is the default remote.   
> It is
> what you can change to fetch from somewhere else _by default_.  So I
> absolutely detest your changes in that regard, who help _noboy_ except
> people who like confusion.
>
> So just change remote.origin.url, but do not mess up our nice  
> source code.
>
> If your issue is with git-submodule, then fix _that_, and do not "fix"
> _anything_ else.

If I understood Mark correctly, this would not solve his problem
because he wants to *avoid* origin; and his arguments make sense
to me.  Unfortunately I cannot contribute any real world
experience, because I chose to abandoned the idea of using
submodules.

However, this does not mean that I'll never use submodules.
Actually I am pretty sure submodules could become very helpful
end of March; and I should also support multiple servers at that
time.  So I would be happy if the use case described by Mark was
well supported by git.

	Steffen

Re: [PATCH 0 of 7] [resend] - Improve handling remotes, origin, submodules

[Johannes Schindelin]

Hi,

On Mon, 4 Feb 2008, Steffen Prohaska wrote:

> On Feb 3, 2008, at 11:43 PM, Johannes Schindelin wrote:
> 
> > If your issue is with git-submodule, then fix _that_, and do not "fix" 
> > _anything_ else.
> 
> If I understood Mark correctly, this would not solve his problem
> because he wants to *avoid* origin;

So he should *avoid* origin.  Not redefine it.

For example, if you want to update your submodules from their respective 
"bollywog" remotes, it is perfectly fine with me to introduce a notion

	$ git submodule update --from bollywog

If you do not want to fetch from here one time, and from there the next, 
then there is _simply_ _no_ _point_ in pointing to another remote _name_.

You cannot put multiple urls into .gitmodules _anyway_, so they _have_ to 
be modified locally, and then the people can look up their remote urls.

Putting in another layer of indirection, which can very well be 
unsynchronised is what I am strongly in "unfavour" of.

Frankly, I am puzzled how long this discussion drags on.  If I would not 
be concerned about the loss of simplicity of git's core _and concepts_, I 
would not even bother to respond.

Ciao,
Dscho