[PATCH 0/5] misc. submodule related changes

[PATCH 0/5] misc. submodule related changes

[Lars Hjemli]

Here is a reworked patch-series for git-submodule, trying to cater for
the issues with the previous series.

Shortlog:
 [1/5] t7400: barf if git-submodule removes or replaces a file
 [2/5] git-submodule: remember to checkout after clone
 [3/5] Rename sections from "module" to "submodule" in .gitmodules
 [4/5] git-submodule: give submodules proper names
 [5/5] Add gitmodules(5)

Diffstat:
Documentation/Makefile       |    2 +-
Documentation/gitmodules.txt |   63 ++++++++++++++++++++++++++++++++++++++++++
git-submodule.sh             |   52 +++++++++++++++++++++++-----------
t/t7400-submodule-basic.sh   |   22 +++++++++++---
4 files changed, 116 insertions(+), 23 deletions(-)

[PATCH 1/5] t7400: barf if git-submodule removes or replaces a file

[Lars Hjemli]

The test for an unmolested file wouldn't fail properly if the file had been
removed or replaced by something other than a regular file. This fixes it.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 t/t7400-submodule-basic.sh |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 3940433..74fafce 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -72,7 +72,7 @@ test_expect_success 'update should fail when path is used by a file' '
 	then
 		echo "[OOPS] update should have failed"
 		false
-	elif test -f lib && test "$(cat lib)" != "hello"
+	elif test "$(cat lib)" != "hello"
 	then
 		echo "[OOPS] update failed but lib file was molested"
 		false
-- 
1.5.2.1.914.gbd3a7

[PATCH 2/5] git-submodule: remember to checkout after clone

[Lars Hjemli]

After the initial clone of a submodule, no files would be checked out in
the submodule directory if the submodule HEAD was equal to the SHA-1
specified in the index of the containing repository. This fixes the problem
by simply ignoring submodule HEAD for a fresh clone.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 git-submodule.sh |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 8bdd99a..4a6d64d 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -100,12 +100,13 @@ modules_update()
 		if ! test -d "$path"/.git
 		then
 			module_clone "$path" "$url" || exit
+			subsha1=
+		else
+			subsha1=$(unset GIT_DIR && cd "$path" &&
+				git-rev-parse --verify HEAD) ||
+			die "Unable to find current revision of submodule '$path'"
 		fi
 
-		subsha1=$(unset GIT_DIR && cd "$path" &&
-			git-rev-parse --verify HEAD) ||
-		die "Unable to find current revision of submodule '$path'"
-
 		if test "$subsha1" != "$sha1"
 		then
 			(unset GIT_DIR && cd "$path" && git-fetch &&
-- 
1.5.2.1.914.gbd3a7

[PATCH 3/5] Rename sections from "module" to "submodule" in .gitmodules

[Lars Hjemli]

On 6/10/07, Junio C Hamano <gitster@pobox.com> wrote:
> "Lars Hjemli" <hjemli@gmail.com> writes:
>
> > Hmm, maybe I should just rename [module] to [submodule] right now? It
> > would be better forward compatible with the proposed extension, it
> > would 'harmonize' the section names used in .gitmodules and
> > .git/config, and it would offer a clean break from what's currently
> > supported in 'master'.
>
> Yes, the difference between '[submodule]' vs '[module]' in
> .git/config and .gitmodules confused me while looking at your
> latest patch series.  I am in favor of unifying them.  We would
> not be breaking any released version if we harmonize them now.

Here it is.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 git-submodule.sh           |    2 +-
 t/t7400-submodule-basic.sh |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 4a6d64d..6c83c52 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -66,7 +66,7 @@ modules_init()
 		url=$(git-config submodule."$path".url)
 		test -z "$url" || continue
 
-		url=$(GIT_CONFIG=.gitmodules git-config module."$path".url)
+		url=$(GIT_CONFIG=.gitmodules git-config submodule."$path".url)
 		test -z "$url" &&
 		die "No url found for submodule '$path' in .gitmodules"
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 74fafce..9f2d4f9 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -40,7 +40,7 @@ test_expect_success 'Prepare submodule testing' '
 	git-add a lib z &&
 	git-commit -m "super commit 1" &&
 	mv lib .subrepo &&
-	GIT_CONFIG=.gitmodules git-config module.lib.url git://example.com/lib.git
+	GIT_CONFIG=.gitmodules git-config submodule.lib.url git://example.com/lib.git
 '
 
 test_expect_success 'status should only print one line' '
-- 
1.5.2.1.914.gbd3a7

[PATCH 4/5] git-submodule: give submodules proper names

[Lars Hjemli]

This changes the way git-submodule uses .gitmodules: Subsections no longer
specify the submodule path, they now specify the submodule name. The
submodule path is found under the new key "submodule.<name>.path", which is
a required key.

With this change a submodule can be moved between different 'checkout paths'
without upsetting git-submodule.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 git-submodule.sh           |   45 ++++++++++++++++++++++++++++++-------------
 t/t7400-submodule-basic.sh |   20 +++++++++++++++---
 2 files changed, 47 insertions(+), 18 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 6c83c52..89a3885 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -25,6 +25,19 @@ say()
 	fi
 }
 
+#
+# Map submodule path to submodule name
+#
+# $1 = path
+#
+module_name()
+{
+       name=$(GIT_CONFIG=.gitmodules git-config --get-regexp '^submodule\..*\.path$' "$1" |
+       sed -nre 's/^submodule\.(.+)\.path .+$/\1/p')
+       test -z "$name" &&
+       die "No submodule mapping found in .gitmodules for path '$path'"
+       echo "$name"
+}
 
 #
 # Clone a submodule
@@ -49,7 +62,7 @@ module_clone()
 	die "A file already exist at path '$path'"
 
 	git-clone -n "$url" "$path" ||
-	die "Clone of submodule '$path' failed"
+	die "Clone of '$url' into submodule path '$path' failed"
 }
 
 #
@@ -63,17 +76,18 @@ modules_init()
 	while read mode sha1 stage path
 	do
 		# Skip already registered paths
-		url=$(git-config submodule."$path".url)
+		name=$(module_name "$path") || exit
+		url=$(git-config submodule."$name".url)
 		test -z "$url" || continue
 
-		url=$(GIT_CONFIG=.gitmodules git-config submodule."$path".url)
+		url=$(GIT_CONFIG=.gitmodules git-config submodule."$name".url)
 		test -z "$url" &&
-		die "No url found for submodule '$path' in .gitmodules"
+		die "No url found for submodule path '$path' in .gitmodules"
 
-		git-config submodule."$path".url "$url" ||
-		die "Failed to register url for submodule '$path'"
+		git-config submodule."$name".url "$url" ||
+		die "Failed to register url for submodule path '$path'"
 
-		say "Submodule '$path' registered with url '$url'"
+		say "Submodule '$name' ($url) registered for path '$path'"
 	done
 }
 
@@ -87,13 +101,14 @@ modules_update()
 	git ls-files --stage -- "$@" | grep -e '^160000 ' |
 	while read mode sha1 stage path
 	do
-		url=$(git-config submodule."$path".url)
+		name=$(module_name "$path") || exit
+		url=$(git-config submodule."$name".url)
 		if test -z "$url"
 		then
 			# Only mention uninitialized submodules when its
 			# path have been specified
 			test "$#" != "0" &&
-			say "Submodule '$path' not initialized"
+			say "Submodule path '$path' not initialized"
 			continue
 		fi
 
@@ -104,22 +119,22 @@ modules_update()
 		else
 			subsha1=$(unset GIT_DIR && cd "$path" &&
 				git-rev-parse --verify HEAD) ||
-			die "Unable to find current revision of submodule '$path'"
+			die "Unable to find current revision in submodule path '$path'"
 		fi
 
 		if test "$subsha1" != "$sha1"
 		then
 			(unset GIT_DIR && cd "$path" && git-fetch &&
 				git-checkout -q "$sha1") ||
-			die "Unable to checkout '$sha1' in submodule '$path'"
+			die "Unable to checkout '$sha1' in submodule path '$path'"
 
-			say "Submodule '$path': checked out '$sha1'"
+			say "Submodule path '$path': checked out '$sha1'"
 		fi
 	done
 }
 
 #
-# List all registered submodules, prefixed with:
+# List all submodules, prefixed with:
 #  - submodule not initialized
 #  + different revision checked out
 #
@@ -133,7 +148,9 @@ modules_list()
 	git ls-files --stage -- "$@" | grep -e '^160000 ' |
 	while read mode sha1 stage path
 	do
-		if ! test -d "$path"/.git
+		name=$(module_name "$path") || exit
+		url=$(git-config submodule."$name".url)
+		if test -z "url" || ! test -d "$path"/.git
 		then
 			say "-$sha1 $path"
 			continue;
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 9f2d4f9..7a9b505 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -18,7 +18,7 @@ subcommands of git-submodule.
 #  -add directory lib to 'superproject', this creates a DIRLINK entry
 #  -add a couple of regular files to enable testing of submodule filtering
 #  -mv lib subrepo
-#  -add an entry to .gitmodules for path 'lib'
+#  -add an entry to .gitmodules for submodule 'example'
 #
 test_expect_success 'Prepare submodule testing' '
 	mkdir lib &&
@@ -40,7 +40,19 @@ test_expect_success 'Prepare submodule testing' '
 	git-add a lib z &&
 	git-commit -m "super commit 1" &&
 	mv lib .subrepo &&
-	GIT_CONFIG=.gitmodules git-config submodule.lib.url git://example.com/lib.git
+	GIT_CONFIG=.gitmodules git-config submodule.example.url git://example.com/lib.git
+'
+
+test_expect_success 'status should fail for unmapped paths' '
+	if git-submodule status
+	then
+		echo "[OOPS] submodule status succeeded"
+		false
+	elif ! GIT_CONFIG=.gitmodules git-config submodule.example.path lib
+	then
+		echo "[OOPS] git-config failed to update .gitmodules"
+		false
+	fi
 '
 
 test_expect_success 'status should only print one line' '
@@ -54,12 +66,12 @@ test_expect_success 'status should initially be "missing"' '
 
 test_expect_success 'init should register submodule url in .git/config' '
 	git-submodule init &&
-	url=$(git-config submodule.lib.url) &&
+	url=$(git-config submodule.example.url) &&
 	if test "$url" != "git://example.com/lib.git"
 	then
 		echo "[OOPS] init succeeded but submodule url is wrong"
 		false
-	elif ! git-config submodule.lib.url ./.subrepo
+	elif ! git-config submodule.example.url ./.subrepo
 	then
 		echo "[OOPS] init succeeded but update of url failed"
 		false
-- 
1.5.2.1.914.gbd3a7

[PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

This adds documentation for the .gitmodules file.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 Documentation/Makefile       |    2 +-
 Documentation/gitmodules.txt |   62 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 1 deletions(-)
 create mode 100644 Documentation/gitmodules.txt

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 9cef480..2ad18e0 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -2,7 +2,7 @@ MAN1_TXT= \
 	$(filter-out $(addsuffix .txt, $(ARTICLES) $(SP_ARTICLES)), \
 		$(wildcard git-*.txt)) \
 	gitk.txt
-MAN5_TXT=gitattributes.txt gitignore.txt
+MAN5_TXT=gitattributes.txt gitignore.txt gitmodules.txt
 MAN7_TXT=git.txt
 
 DOC_HTML=$(patsubst %.txt,%.html,$(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT))
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
new file mode 100644
index 0000000..6c7d9bf
--- /dev/null
+++ b/Documentation/gitmodules.txt
@@ -0,0 +1,62 @@
+gitmodules(5)
+=============
+
+NAME
+----
+gitmodules - defining submodule properties
+
+SYNOPSIS
+--------
+.gitmodules
+
+
+DESCRIPTION
+-----------
+
+The `.gitmodules` file, located in the top-level directory of a git
+working tree, is a text file with a syntax matching the requirements
+of gitlink:git-config[1].
+
+The file contains one subsection per submodule, and the subsection value
+is the name of the submodule. Each submodule section also contains the
+following required keys:
+
+submodule.<name>.path::
+	Defines the path, relative to the top-level directory of the git
+	working tree, where the submodule is expected to be checked out.
+	The path name must not end with a `/`. All submodule paths must
+	be unique within the .gitmodules file.
+
+submodule.<name>.url::
+	Defines an url from where the submodule repository can be cloned.
+
+
+EXAMPLES
+--------
+
+Consider the following .gitmodules file:
+
+	[submodule 'libfoo']
+		path = include/foo
+		url = git://foo.com/git/lib.git
+
+	[submodule 'libbar']
+		path = include/bar
+		url = git://bar.com/git/lib.git
+
+
+This defines two submodules, `libfoo` and `libbar`. These are expected to
+be checked out in the paths 'include/foo' and 'include/bar', and for both
+submodules an url is specified which can be used for cloning the submodules.
+
+SEE ALSO
+--------
+gitlink:git-submodule[1] gitlink:git-config[1]
+
+DOCUMENTATION
+-------------
+Documentation by Lars Hjemli <hjemli@gmail.com>
+
+GIT
+---
+Part of the gitlink:git[7] suite
-- 
1.5.2.1.914.gbd3a7

Re: [PATCH 5/5] Add gitmodules(5)

[Frank Lichtenheld]

On Mon, Jun 11, 2007 at 09:12:25PM +0200, Lars Hjemli wrote:
> +Consider the following .gitmodules file:
> +
> +	[submodule 'libfoo']
> +		path = include/foo
> +		url = git://foo.com/git/lib.git
> +
> +	[submodule 'libbar']
> +		path = include/bar
> +		url = git://bar.com/git/lib.git
> +

Still the wrong quotes.

Gruesse,
-- 
Frank Lichtenheld <frank@lichtenheld.de>
www: http://www.djpig.de/

[PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

This adds documentation for the .gitmodules file.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---

On 6/12/07, Frank Lichtenheld <frank@lichtenheld.de> wrote:
> On Mon, Jun 11, 2007 at 09:12:25PM +0200, Lars Hjemli wrote:
> > +Consider the following .gitmodules file:
> > +
> > +     [submodule 'libfoo']
> > +             path = include/foo
> > +             url = git://foo.com/git/lib.git
> > +
> > +     [submodule 'libbar']
> > +             path = include/bar
> > +             url = git://bar.com/git/lib.git
> > +
> 
> Still the wrong quotes.

Thanks for noticing


 Documentation/Makefile       |    2 +-
 Documentation/gitmodules.txt |   62 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 1 deletions(-)
 create mode 100644 Documentation/gitmodules.txt

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 9cef480..2ad18e0 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -2,7 +2,7 @@ MAN1_TXT= \
 	$(filter-out $(addsuffix .txt, $(ARTICLES) $(SP_ARTICLES)), \
 		$(wildcard git-*.txt)) \
 	gitk.txt
-MAN5_TXT=gitattributes.txt gitignore.txt
+MAN5_TXT=gitattributes.txt gitignore.txt gitmodules.txt
 MAN7_TXT=git.txt
 
 DOC_HTML=$(patsubst %.txt,%.html,$(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT))
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
new file mode 100644
index 0000000..6c7d9bf
--- /dev/null
+++ b/Documentation/gitmodules.txt
@@ -0,0 +1,62 @@
+gitmodules(5)
+=============
+
+NAME
+----
+gitmodules - defining submodule properties
+
+SYNOPSIS
+--------
+.gitmodules
+
+
+DESCRIPTION
+-----------
+
+The `.gitmodules` file, located in the top-level directory of a git
+working tree, is a text file with a syntax matching the requirements
+of gitlink:git-config[1].
+
+The file contains one subsection per submodule, and the subsection value
+is the name of the submodule. Each submodule section also contains the
+following required keys:
+
+submodule.<name>.path::
+	Defines the path, relative to the top-level directory of the git
+	working tree, where the submodule is expected to be checked out.
+	The path name must not end with a `/`. All submodule paths must
+	be unique within the .gitmodules file.
+
+submodule.<name>.url::
+	Defines an url from where the submodule repository can be cloned.
+
+
+EXAMPLES
+--------
+
+Consider the following .gitmodules file:
+
+	[submodule "libfoo"]
+		path = include/foo
+		url = git://foo.com/git/lib.git
+
+	[submodule "libbar"]
+		path = include/bar
+		url = git://bar.com/git/lib.git
+
+
+This defines two submodules, `libfoo` and `libbar`. These are expected to
+be checked out in the paths 'include/foo' and 'include/bar', and for both
+submodules an url is specified which can be used for cloning the submodules.
+
+SEE ALSO
+--------
+gitlink:git-submodule[1] gitlink:git-config[1]
+
+DOCUMENTATION
+-------------
+Documentation by Lars Hjemli <hjemli@gmail.com>
+
+GIT
+---
+Part of the gitlink:git[7] suite
-- 
1.5.2.1.914.gbd3a7

Re: [PATCH 5/5] Add gitmodules(5)

[Sven Verdoolaege]

On Tue, Jun 12, 2007 at 09:05:21AM +0200, Lars Hjemli wrote:
> +The file contains one subsection per submodule, and the subsection value
> +is the name of the submodule. Each submodule section also contains the
> +following required keys:

Your only argument for making them required was that Junio thought that
not having them might be ambiguous, but he doesn't think so anymore.

> +submodule.<name>.path::
> +	Defines the path, relative to the top-level directory of the git

Your previous patch had "_a_ path" instead of "_the_ path".
I prefer the former since it allows a module to be checkoud out
at multiple locations.

skimo

Re: [PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

On 6/12/07, Sven Verdoolaege <skimo@kotnet.org> wrote:
> On Tue, Jun 12, 2007 at 09:05:21AM +0200, Lars Hjemli wrote:
> > +The file contains one subsection per submodule, and the subsection value
> > +is the name of the submodule. Each submodule section also contains the
> > +following required keys:
>
> Your only argument for making them required was that Junio thought that
> not having them might be ambiguous, but he doesn't think so anymore.

Well, I actually also had an argument about being strict in the
beginning and possibly loosen the rules later on. Anyways, I can do a
patch on top of this series (if/when it's accepted) and let Junio
decide to apply or drop the 'optional path' stuff.

>
> > +submodule.<name>.path::
> > +     Defines the path, relative to the top-level directory of the git
>
> Your previous patch had "_a_ path" instead of "_the_ path".
> I prefer the former since it allows a module to be checkoud out
> at multiple locations.

This is somewhat intentional. I want to move the submodule repos into
.git/submodules/$name/ (with working dir) and symlink this directory
when 'checking out' the submodule. This would be a simple solution for
the following problems:
  -keeping submodule modifications between checkouts
  -having submodules within submodules

Multiple checkout paths for a single submodule will bring havoc on
this plan, so I need to ask: what is the use-case for multiple
checkout paths?

-- 
larsh

Re: [PATCH 5/5] Add gitmodules(5)

[Josef Weidendorfer]

On Tuesday 12 June 2007, Lars Hjemli wrote:
> This is somewhat intentional. I want to move the submodule repos into
> .git/submodules/$name/ (with working dir) and symlink this directory
> when 'checking out' the submodule. This would be a simple solution for
> the following problems:
>   -keeping submodule modifications between checkouts
>   -having submodules within submodules

Interesting idea.

How does this work
(1) if the submodule checkout changes with the supermodule checkout?
You still would have to store the modifications somewhere.
(2) on platforms which do not allow symlinks

A workaround for problem (1) would be to create multiple checkouts of the
same submodule if modified, e.g. in .git/submodule/$name/$sha1 .
 
Allowing people to work like that is nice, but it should not be forced.
It would also be nice to allow the user to specify another place where
submodule checkouts are to be stored, e.g. when multiple supermodules
share the same submodule.

Josef

> 
> Multiple checkout paths for a single submodule will bring havoc on
> this plan, so I need to ask: what is the use-case for multiple
> checkout paths?
> 

Re: [PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

On 6/12/07, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> On Tuesday 12 June 2007, Lars Hjemli wrote:
> > This is somewhat intentional. I want to move the submodule repos into
> > .git/submodules/$name/ (with working dir) and symlink this directory
> > when 'checking out' the submodule. This would be a simple solution for
> > the following problems:
> >   -keeping submodule modifications between checkouts
> >   -having submodules within submodules
>
> Interesting idea.
>
> How does this work
> (1) if the submodule checkout changes with the supermodule checkout?
> You still would have to store the modifications somewhere.

If you're thinking about the detached HEAD: yeah, that's a problem. My
initial plan (with later modifications) was something like this:

[path "lib1"]
  submodule=lib
  branch=stable

[path "lib2"]
  submodule=lib
  branch=bleeding

[submodule "lib"]
  url=git://example.com/lib.git

$ git-submodule init
  git-config submodule.lib.url git://example.com/lib.git

$ git-submodule update
  git-clone --bare git://example.com/lib.git .git/submodules/lib.git
  git-clone -l -s -n .git/submodules/lib.git lib1
  (cd lib1 && git-checkout $sha1)
  git-clone -l -s -n .git/submodules/lib.git lib2
  (cd lib2 && git-checkout $sha2)

git-submodule push:
  (cd lib1 && git-push origin $branch1)
  (cd lib2 && git-push origin $branch2)

I thought I could avoid 'git-submodule push' by using symlinks, but
you're right. It will not work. Back to the drawing board (again...)


> (2) on platforms which do not allow symlinks

Ok, bad idea.


>
> A workaround for problem (1) would be to create multiple checkouts of the
> same submodule if modified, e.g. in .git/submodule/$name/$sha1 .

And the $sha1 would be the sha1 found in the index? I don't think this
would work either. If two branches in the superproject checkout the
same submodule sha1, you could possibly want to keep different changes
in the submodule depending on which branch of the superproject is
checked out.

I guess the user will have to both commit and push submodule changes
before switching branches etc. But that might not be too bad, at least
for the initial submodule support.

>
> Allowing people to work like that is nice, but it should not be forced.
> It would also be nice to allow the user to specify another place where
> submodule checkouts are to be stored, e.g. when multiple supermodules
> share the same submodule.

True. Maybe submodule.<name>.repopath in .git/config? (If not
specified, default to .git/submodules/<name>.git)

--
larsh

Re: [PATCH 5/5] Add gitmodules(5)

[Josef Weidendorfer]

On Tuesday 12 June 2007, Lars Hjemli wrote:
> If you're thinking about the detached HEAD: yeah, that's a problem. My
> initial plan (with later modifications) was something like this:
> 
> ...
> 
> $ git-submodule update
>   git-clone --bare git://example.com/lib.git .git/submodules/lib.git
>   git-clone -l -s -n .git/submodules/lib.git lib1
>   (cd lib1 && git-checkout $sha1)
>   git-clone -l -s -n .git/submodules/lib.git lib2
>   (cd lib2 && git-checkout $sha2)
> ...

That looks fine to me.
 
> git-submodule push:
>   (cd lib1 && git-push origin $branch1)
>   (cd lib2 && git-push origin $branch2)

This could be problematic. You are only storing changes on the
given branch. Ah, you wanted to avoid this problem with the symlink?

Perhaps we need better support for "push all reachable objects
which are not on the remote side together with any branch tip changes".
Or is this somehow already available with git-push?

Or ...

> I thought I could avoid 'git-submodule push' by using symlinks, but
> you're right. It will not work. Back to the drawing board (again...)

... we revive the "lightweight checkout" idea: share everything
but index and HEAD with a given repository. Similar to
"contrib/workdir/git-new-workdir" but witt support in git-core
to avoid the need for symlinks.

> > A workaround for problem (1) would be to create multiple checkouts of the
> > same submodule if modified, e.g. in .git/submodule/$name/$sha1 .
> 
> And the $sha1 would be the sha1 found in the index? I don't think this
> would work either. If two branches in the superproject checkout the
> same submodule sha1, you could possibly want to keep different changes
> in the submodule depending on which branch of the superproject is
> checked out.

OK, does not work.

> > Allowing people to work like that is nice, but it should not be forced.
> > It would also be nice to allow the user to specify another place where
> > submodule checkouts are to be stored, e.g. when multiple supermodules
> > share the same submodule.
> 
> True. Maybe submodule.<name>.repopath in .git/config? (If not
> specified, default to .git/submodules/<name>.git)

Sounds good. However, can be done later.

Josef

> 
> --
> larsh
>

Re: [PATCH 5/5] Add gitmodules(5)

[Sven Verdoolaege]

On Tue, Jun 12, 2007 at 10:27:00AM +0200, Lars Hjemli wrote:
> On 6/12/07, Sven Verdoolaege <skimo@kotnet.org> wrote:
> >Your previous patch had "_a_ path" instead of "_the_ path".
> >I prefer the former since it allows a module to be checkoud out
> >at multiple locations.
> 
> This is somewhat intentional. I want to move the submodule repos into
> .git/submodules/$name/ (with working dir) and symlink this directory

I had that in my patch series, but I got a complaint that symlinks
don't work on Windows.

> Multiple checkout paths for a single submodule will bring havoc on
> this plan, so I need to ask: what is the use-case for multiple
> checkout paths?

The case where you need two different versions of the same
submodule in one (presumably big) project.
(Not that I need that just now.)

skimo

Re: [PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

On 6/12/07, Sven Verdoolaege <skimo@kotnet.org> wrote:
> On Tue, Jun 12, 2007 at 10:27:00AM +0200, Lars Hjemli wrote:
> > On 6/12/07, Sven Verdoolaege <skimo@kotnet.org> wrote:
> > >Your previous patch had "_a_ path" instead of "_the_ path".
> > >I prefer the former since it allows a module to be checkoud out
> > >at multiple locations.
> >
> > This is somewhat intentional. I want to move the submodule repos into
> > .git/submodules/$name/ (with working dir) and symlink this directory
>
> I had that in my patch series, but I got a complaint that symlinks
> don't work on Windows.

Yeah, I didn't consider windows.

>
> > Multiple checkout paths for a single submodule will bring havoc on
> > this plan, so I need to ask: what is the use-case for multiple
> > checkout paths?
>
> The case where you need two different versions of the same
> submodule in one (presumably big) project.

Let me rephrase: why would anyone need to checkout two different
versions of the same submodule simultaneously inside a single
superproject?

-- 
larsh

Re: [PATCH 5/5] Add gitmodules(5)

[Johannes Sixt]

Lars Hjemli wrote:
> 
> On 6/12/07, Sven Verdoolaege <skimo@kotnet.org> wrote:
> > On Tue, Jun 12, 2007 at 09:05:21AM +0200, Lars Hjemli wrote:
> > > +submodule.<name>.path::
> > > +     Defines the path, relative to the top-level directory of the git
> >
> > Your previous patch had "_a_ path" instead of "_the_ path".
> > I prefer the former since it allows a module to be checkoud out
> > at multiple locations.
> 
> This is somewhat intentional. I want to move the submodule repos into
> .git/submodules/$name/ (with working dir) and symlink this directory
> when 'checking out' the submodule. This would be a simple solution for
> the following problems:
>   -keeping submodule modifications between checkouts
>   -having submodules within submodules

It has already been said in the past that symlinks are *bad*. The don't
exist on Windows (MinGW). Please do not use symlinks for such central
concepts.

-- Hannes

Re: [PATCH 5/5] Add gitmodules(5)

[Johannes Sixt]

Lars Hjemli wrote:
> Multiple checkout paths for a single submodule will bring havoc on
> this plan, so I need to ask: what is the use-case for multiple
> checkout paths?

A use-case is the admin directory in the KDE repository. It has:

KDE (superproject)
 +- kdelibs (subproject)
 |   +- admin (subproject)
 |   +- subdir1
 |   +- ...
 +- kdebase (subproject)
 |   +- admin (subproject)
 |   +- subdir2
 |   +- ...
 +- kdenetwork (subproject)
 |   +- admin (subproject)
 |   +- subdir3
 |   +- ...
 ...

-- Hannes

[PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

(readded the gitlist)

On 6/12/07, Johannes Sixt <J.Sixt@eudaptics.com> wrote:
> Lars Hjemli wrote:
> > Multiple checkout paths for a single submodule will bring havoc on
> > this plan, so I need to ask: what is the use-case for multiple
> > checkout paths?
>
> A use-case is the admin directory in the KDE repository. It has:
>
> KDE (superproject)
>  +- kdelibs (subproject)
>  |   +- admin (subproject)
>  |   +- subdir1
>  |   +- ...
>  +- kdebase (subproject)
>  |   +- admin (subproject)
>  |   +- subdir2
>  |   +- ...
>  +- kdenetwork (subproject)
>  |   +- admin (subproject)
>  |   +- subdir3
>  |   +- ...
>  ...

But in this case, 'admin' isn't a submodule/subproject contained by
KDE, right? It's contained in three different submodules/subprojects:
kdelibs, kdebase and kdenetwork.

--
larsh

Re: [PATCH 5/5] Add gitmodules(5)

[Johannes Sixt]

Lars Hjemli wrote:
> 
> (readded the gitlist)
> 
> On 6/12/07, Johannes Sixt <J.Sixt@eudaptics.com> wrote:
> > Lars Hjemli wrote:
> > > Multiple checkout paths for a single submodule will bring havoc on
> > > this plan, so I need to ask: what is the use-case for multiple
> > > checkout paths?
> >
> > A use-case is the admin directory in the KDE repository. It has:
> >
> > KDE (superproject)
> >  +- kdelibs (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir1
> >  |   +- ...
> >  +- kdebase (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir2
> >  |   +- ...
> >  +- kdenetwork (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir3
> >  |   +- ...
> >  ...
> 
> But in this case, 'admin' isn't a submodule/subproject contained by
> KDE, right? It's contained in three different submodules/subprojects:
> kdelibs, kdebase and kdenetwork.

Notice how kdelibs, kdebase and kdenetwork are both submodule and
supermodule: They host the submodule admin and are hosted by KDE.

(If I missed the point, then it's because I didn't follow the
discussion; I jumped in because I noticed the symlink proposal by
chance.)

-- Hannes

Re: [PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

On 6/12/07, Johannes Sixt <J.Sixt@eudaptics.com> wrote:
> Lars Hjemli wrote:
> >
> > (readded the gitlist)
> >
> > On 6/12/07, Johannes Sixt <J.Sixt@eudaptics.com> wrote:
> > > Lars Hjemli wrote:
> > > > Multiple checkout paths for a single submodule will bring havoc on
> > > > this plan, so I need to ask: what is the use-case for multiple
> > > > checkout paths?
> > >
> > > A use-case is the admin directory in the KDE repository. It has:
> > >
> > > KDE (superproject)
> > >  +- kdelibs (subproject)
> > >  |   +- admin (subproject)
> > >  |   +- subdir1
> > >  |   +- ...
> > >  +- kdebase (subproject)
> > >  |   +- admin (subproject)
> > >  |   +- subdir2
> > >  |   +- ...
> > >  +- kdenetwork (subproject)
> > >  |   +- admin (subproject)
> > >  |   +- subdir3
> > >  |   +- ...
> > >  ...
> >
> > But in this case, 'admin' isn't a submodule/subproject contained by
> > KDE, right? It's contained in three different submodules/subprojects:
> > kdelibs, kdebase and kdenetwork.
>
> Notice how kdelibs, kdebase and kdenetwork are both submodule and
> supermodule: They host the submodule admin and are hosted by KDE.
>

Exactly my point ;-)

> (If I missed the point, then it's because I didn't follow the
> discussion; I jumped in because I noticed the symlink proposal by
> chance.)

In this case, I would assume that e.g. kdelibs contain a submodule
entry for path admin in its index, accompanied by a .gitmodules file
saying that the path admin is mapped to this or that submodule. I
would not expect the KDE 'supersuperproject' to know about admin at
all, neither in its index nor .gitmodules.

--
larsh

Re: [PATCH 5/5] Add gitmodules(5)

[Josef Weidendorfer]

On Tuesday 12 June 2007, Lars Hjemli wrote:
> On 6/12/07, Johannes Sixt <J.Sixt@eudaptics.com> wrote:

> > > > KDE (superproject)
> > > >  +- kdelibs (subproject)
> > > >  |   +- admin (subproject)
> > > >  |   +- subdir1
> > > >  |   +- ...
> > > >  +- kdebase (subproject)
> > > >  |   +- admin (subproject)
> > > >  |   +- subdir2
> > > >  |   +- ...
> > > >  +- kdenetwork (subproject)
> > > >  |   +- admin (subproject)
> > > >  |   +- subdir3
> > > >  |   +- ...
> > > >  ...
> 
> In this case, I would assume that e.g. kdelibs contain a submodule
> entry for path admin in its index, accompanied by a .gitmodules file
> saying that the path admin is mapped to this or that submodule.

Exactly. That's just the "submodule" in "submodule" case we should
support, too.

> I 
> would not expect the KDE 'supersuperproject' to know about admin at
> all, neither in its index nor .gitmodules.

The admin submodule contains KDE specific things. So of course it
also would be a submodule in the grand whole KDE superduper module.
But that does not really matter.

However, to not have a lot of copies of the admin submodule
in

 .git/submodule/admin
 .git/submodule/kdelibs/.git/submodule/admin
 .git/submodule/kdebase/.git/submodule/admin
 .git/submodule/kdenetwork/.git/submodule/admin

the just suggested submodule.<name>.repopath to specify a repository
outside of .git/submodule to be shared by kdelibs,kdebase,... would
be fine.

Josef

Re: [PATCH 5/5] Add gitmodules(5)

[Lars Hjemli]

On 6/12/07, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> On Tuesday 12 June 2007, Lars Hjemli wrote:
> > I
> > would not expect the KDE 'supersuperproject' to know about admin at
> > all, neither in its index nor .gitmodules.
>
> The admin submodule contains KDE specific things. So of course it
> also would be a submodule in the grand whole KDE superduper module.

Aha! But as you say:

> But that does not really matter.

Exactly.

>
> However, to not have a lot of copies of the admin submodule
> in
>
>  .git/submodule/admin
>  .git/submodule/kdelibs/.git/submodule/admin
>  .git/submodule/kdebase/.git/submodule/admin
>  .git/submodule/kdenetwork/.git/submodule/admin
>
> the just suggested submodule.<name>.repopath to specify a repository
> outside of .git/submodule to be shared by kdelibs,kdebase,... would
> be fine.

Yes, repopath would work out nice for KDE (which btw seems to be a
great test-case for git-submodule)

--
larsh

Re: [PATCH 5/5] Add gitmodules(5)

[Johannes Sixt]

Josef Weidendorfer wrote:
> However, to not have a lot of copies of the admin submodule
> in
> 
>  .git/submodule/admin
>  .git/submodule/kdelibs/.git/submodule/admin
>  .git/submodule/kdebase/.git/submodule/admin
>  .git/submodule/kdenetwork/.git/submodule/admin
> 
> the just suggested submodule.<name>.repopath to specify a repository
> outside of .git/submodule to be shared by kdelibs,kdebase,... would
> be fine.

This clearly shows that having the repositories of submodules in
.git/submodule does not buy you enough to avoid duplication.

(I don't see enough reason to place a repo for submodule X in project Y
outside its "natural" checked-out directory in project Y. But then, I
haven't followed the discussion. Please ignore me if above layout choice
is for more than just avoiding duplication.)

-- Hannes

Re: [PATCH 5/5] Add gitmodules(5)

[Josef Weidendorfer]

On Tuesday 12 June 2007, Johannes Sixt wrote:
> Josef Weidendorfer wrote:
> > However, to not have a lot of copies of the admin submodule
> > in
> > 
> >  .git/submodule/admin
> >  .git/submodule/kdelibs/.git/submodule/admin
> >  .git/submodule/kdebase/.git/submodule/admin
> >  .git/submodule/kdenetwork/.git/submodule/admin
> > 
> > the just suggested submodule.<name>.repopath to specify a repository
> > outside of .git/submodule to be shared by kdelibs,kdebase,... would
> > be fine.
> 
> This clearly shows that having the repositories of submodules in
> .git/submodule does not buy you enough to avoid duplication.

The .git/submodule space for sure is not flexible enough for all
possible use cases of submodules (as with KDE repo).
However, as default it seems fine to me.

IMHO sharing of the admin submodule repository should even be possible
if I have a clone of kdelibs and kdebase independent of the big
KDE superproject.

It would be nice to allow submodule.<name>.repopath configs globally
in ~/.gitconfig, and cloning kdelibs should automatically do the
right thing, ie. use the already available admin repo for the kdelibs
clone.

> (I don't see enough reason to place a repo for submodule X in project Y
> outside its "natural" checked-out directory in project Y. But then, I
> haven't followed the discussion.

To easily share the objects, branches and local modifications?

Currently a clone of a superproject always does a full copy of
any submodule databases because it is independent from any other
local git repository; in the KDE case you would get >4 admin copiess
if recursive cloning of submodules inside of submodules does that.

Josef

Re: [PATCH 5/5] Add gitmodules(5)

[Josef Weidendorfer]

On Tuesday 12 June 2007, Josef Weidendorfer wrote:
> IMHO sharing of the admin submodule repository should even be possible
> if I have a clone of kdelibs and kdebase independent of the big
> KDE superproject.
> 
> It would be nice to allow submodule.<name>.repopath configs globally
> in ~/.gitconfig, and cloning kdelibs should automatically do the
> right thing, ie. use the already available admin repo for the kdelibs
> clone.

I just realize that this should be already possible now by setting
submodule.<name>.url in ~/.gitconfig. It could automatically
use "git-clone -l -s -n ..." if the URL is local.

Josef

Re: [PATCH 5/5] Add gitmodules(5)

[Junio C Hamano]

Johannes Sixt <J.Sixt@eudaptics.com> writes:

> Lars Hjemli wrote:
>> Multiple checkout paths for a single submodule will bring havoc on
>> this plan, so I need to ask: what is the use-case for multiple
>> checkout paths?
>
> A use-case is the admin directory in the KDE repository. It has:
>
> KDE (superproject)
>  +- kdelibs (subproject)
>  |   +- admin (subproject)
>  |   +- subdir1
>  |   +- ...
>  +- kdebase (subproject)
>  |   +- admin (subproject)
>  |   +- subdir2
>  |   +- ...
>  +- kdenetwork (subproject)
>  |   +- admin (subproject)
>  |   +- subdir3
>  |   +- ...
>  ...

If these three instances of 'admin' are truly the same project
created in multiple places in the directory hierarchy, what is
the reason that it is not arranged like this instead?

    KDE
     +- admin
     +- kdelibs
     |   +- subdir1
     |   +- ...
     +- kdebase
     |   +- subdir2
     |   +- ...
     +- kdenetwork
     |   +- subdir3
     |   +- ...
     ...

When kdelibs/subdir1 needs to access stuff in admin, instead of
going to ../admin, it could very well go to ../../admin couldn't
it?

It makes me wonder if the KDE's layout you quoted is a good
practice we would want to recommend for other people to follow.
If not, I doubt it is a good idea to model our important concept
after that layout to begin with.

Re: [PATCH 5/5] Add gitmodules(5)

[Sven Verdoolaege]

On Tue, Jun 12, 2007 at 12:03:10PM -0700, Junio C Hamano wrote:
> Johannes Sixt <J.Sixt@eudaptics.com> writes:
> 
> > Lars Hjemli wrote:
> >> Multiple checkout paths for a single submodule will bring havoc on
> >> this plan, so I need to ask: what is the use-case for multiple
> >> checkout paths?
> >
> > A use-case is the admin directory in the KDE repository. It has:
> >
> > KDE (superproject)
> >  +- kdelibs (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir1
> >  |   +- ...
> >  +- kdebase (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir2
> >  |   +- ...
> >  +- kdenetwork (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir3
> >  |   +- ...
> >  ...
> 
> If these three instances of 'admin' are truly the same project
> created in multiple places in the directory hierarchy, what is
> the reason that it is not arranged like this instead?
> 
>     KDE
>      +- admin
>      +- kdelibs
>      |   +- subdir1
>      |   +- ...
>      +- kdebase
>      |   +- subdir2
>      |   +- ...
>      +- kdenetwork
>      |   +- subdir3
>      |   +- ...
>      ...
> 
> When kdelibs/subdir1 needs to access stuff in admin, instead of
> going to ../admin, it could very well go to ../../admin couldn't
> it?

In Hannes' example, kdelibs and kdebase are subprojects of their own.
Surely, you wouldn't want them to depend on something outside of
the (sub)project.

But even in a similar situation where the different parts do not
exist as separate projects, they may advance at a different pace
and therefore need different revisions of the same subproject.

skimo

Re: [PATCH 5/5] Add gitmodules(5)

[Josef Weidendorfer]

On Tuesday 12 June 2007, Junio C Hamano wrote:
> Johannes Sixt <J.Sixt@eudaptics.com> writes:

> > KDE (superproject)
> >  +- kdelibs (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir1
> >  |   +- ...
> >  +- kdebase (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir2
> >  |   +- ...
> >  +- kdenetwork (subproject)
> >  |   +- admin (subproject)
> >  |   +- subdir3
> >  |   +- ...
> >  ...
> 
> If these three instances of 'admin' are truly the same project
> created in multiple places in the directory hierarchy, what is
> the reason that it is not arranged like this instead?
> 
>     KDE
>      +- admin
>      +- kdelibs
>      |   +- subdir1
>      |   +- ...
>      +- kdebase
>      |   +- subdir2
>      |   +- ...
>      +- kdenetwork
>      |   +- subdir3
>      |   +- ...
>      ...

Actually, on the SVN server you have this structure.

KDE applications are put together in groups with
other applications of same kind, e.g. kdenetwork contains
applications for net access.

Now if you want to work on one KDE application e.g. in
kdenetwork, you usually checkout _only_ the kdenetwork
directory. There is no need to have other parts; e.g.
you usually should be able to use the KDE libs from
your distribution - no need to checkout kdelibs.

However, the "admin" directory above contains the build
environment, which has to be checked out so that kdenetwork
is able to build. The applications expect the build tools
to reside in the admin subdirectory.

To checkout admin inside KDE modules such as kdenetwork,
SVN externals are used, which is a primitive form of git
submodules, to automatically checkout the admin directory
on checkout of the KDE module.

So the same admin directory only will be duplicated
multiple times only on the developers side where multiple
KDE modules are checked out.

> When kdelibs/subdir1 needs to access stuff in admin, instead of
> going to ../admin, it could very well go to ../../admin couldn't
> it?

Usually, ../../admin will not exist as explicit checkout on
a developers machine. It would be possible to require this, but
it is much nicer to have each checkout of a KDE module
self-contained, including a copy of admin.

> It makes me wonder if the KDE's layout you quoted is a good
> practice we would want to recommend for other people to follow.
> If not, I doubt it is a good idea to model our important concept
> after that layout to begin with.

In the case of KDE, as far as I remember there is no need to
put _everything_ inside of a the mega supermodule. Every KDE
module (like kdelibs, kdebase, kdenetwork) has its own configure
run and checks dependencies.

What IMHO is an important use case for submodules is to have the
same submodule in multiple different superprojects, as the admin
example shows.

Josef

PS: Above admin example is from KDE3. KDE4 uses an
installed build environment (not really sure).

[PATCH 2/5] git-submodule: remember to checkout after clone

[Lars Hjemli]

After the initial clone of a submodule, no files would be checked out in
the submodule directory if the submodule HEAD was equal to the SHA-1
specified in the index of the containing repository. This fixes the problem
by simply ignoring submodule HEAD for a fresh clone.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
  git-submodule.sh |    9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 8bdd99a..4a6d64d 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -100,12 +100,13 @@ modules_update()
  		if ! test -d "$path"/.git
  		then
  			module_clone "$path" "$url" || exit
+			subsha1=
+		else
+			subsha1=$(unset GIT_DIR && cd "$path" &&
+				git-rev-parse --verify HEAD) ||
+			die "Unable to find current revision of submodule '$path'"
  		fi

-		subsha1=$(unset GIT_DIR && cd "$path" &&
-			git-rev-parse --verify HEAD) ||
-		die "Unable to find current revision of submodule '$path'"
-
  		if test "$subsha1" != "$sha1"
  		then
  			(unset GIT_DIR && cd "$path" && git-fetch &&
-- 
1.5.2.1.914.gbd3a7