[PATCH] Solve git-submodule issues with detached work trees

[PATCH] Solve git-submodule issues with detached work trees

[Daniel Graña]

A common way to track dotfiles with git is using GIT_DIR and
GIT_WORK_TREE to move repository out of ~/.git with something like:

    git init --bare ~/.dotfiles
    alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"

    dotfiles add ~/.bashrc
    dotfiles commit -a -m "add my bashrc"
    ...

but git-submodule complains when trying to add submodules:

    dotfiles submodule add http://path.to/submodule
    fatal: working tree '/home/user' already exists.

    git --git-dir ~/.dotfiles submodule add http://path.to/submodule
    fatal: /usr/lib/git-core/git-submodule cannot be used without a
working tree.

Signed-off-by: Daniel Graña <dangra@gmail.com>
---
 git-submodule.sh                       |    7 +++-
 t/t7409-submodule-detached-worktree.sh |   61 ++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 2 deletions(-)
 create mode 100755 t/t7409-submodule-detached-worktree.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index 5629d87..88ee4ea 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -181,8 +181,11 @@ module_clone()
 		rm -f "$gitdir/index"
 	else
 		mkdir -p "$gitdir_base"
-		git clone $quiet -n ${reference:+"$reference"} \
-			--separate-git-dir "$gitdir" "$url" "$sm_path" ||
+		(
+			clear_local_git_env
+			git clone $quiet -n ${reference:+"$reference"} \
+				--separate-git-dir "$gitdir" "$url" "$sm_path"
+		) ||
 		die "$(eval_gettext "Clone of '\$url' into submodule path
'\$sm_path' failed")"
 	fi

diff --git a/t/t7409-submodule-detached-worktree.sh
b/t/t7409-submodule-detached-worktree.sh
new file mode 100755
index 0000000..db75642
--- /dev/null
+++ b/t/t7409-submodule-detached-worktree.sh
@@ -0,0 +1,61 @@
+#!/bin/sh
+#
+# Copyright (c) 2012 Daniel Graña
+#
+
+test_description='Test submodules on detached working tree
+
+This test verifies that "git submodule" initialization, update and
addition works
+on detahced working trees
+'
+
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+test_expect_success 'submodule on detached working tree' '
+	git init --bare remote &&
+	test_create_repo bundle1 &&
+	(cd bundle1 && test_commit "shoot") &&
+	mkdir home &&
+	(
+		cd home &&
+		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
+		git clone --bare ../remote .dotfiles &&
+		git submodule add ../bundle1 .vim/bundle/sogood &&
+		test_commit "sogood" &&
+		git push origin master
+	) &&
+	mkdir home2 &&
+	(
+		cd home2 &&
+		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
+		git clone --bare ../remote .dotfiles &&
+		git submodule update --init
+	)
+'
+
+test_expect_success 'submodule on detached working pointed by core.worktree' '
+	mkdir home3 &&
+	(
+		cd home3 &&
+		export GIT_DIR="$(pwd)/.dotfiles" &&
+		git clone --bare ../remote "$GIT_DIR" &&
+		git config core.bare false &&
+		git config core.worktree .. &&
+		git submodule add ../bundle1 .vim/bundle/dupe &&
+		test_commit "dupe" &&
+		git push origin master
+	) &&
+	(
+		cd home &&
+		export GIT_DIR="$(pwd)/.dotfiles" &&
+		git config core.bare false &&
+		git config core.worktree .. &&
+		git pull &&
+		git submodule update &&
+		git submodule status &&
+		test -d .vim/bundle/dupe
+	)
+'
+
+test_done
-- 
1.7.5.4

Re: [PATCH] Solve git-submodule issues with detached work trees

[Junio C Hamano]

Daniel Graña <dangra@gmail.com> writes:

> A common way to track dotfiles with git is using GIT_DIR and
> GIT_WORK_TREE to move repository out of ~/.git with something like:
>
>     git init --bare ~/.dotfiles
>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>
>     dotfiles add ~/.bashrc
>     dotfiles commit -a -m "add my bashrc"
>     ...
>
> but git-submodule complains when trying to add submodules:
>
>     dotfiles submodule add http://path.to/submodule
>     fatal: working tree '/home/user' already exists.
>
>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
> working tree.
>
> Signed-off-by: Daniel Graña <dangra@gmail.com>
> ---

I think this is in line with what we discussed earlier on list when
the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
the last time.  Jens?

>  git-submodule.sh                       |    7 +++-
>  t/t7409-submodule-detached-worktree.sh |   61 ++++++++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+), 2 deletions(-)
>  create mode 100755 t/t7409-submodule-detached-worktree.sh
>
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 5629d87..88ee4ea 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -181,8 +181,11 @@ module_clone()
>  		rm -f "$gitdir/index"
>  	else
>  		mkdir -p "$gitdir_base"
> -		git clone $quiet -n ${reference:+"$reference"} \
> -			--separate-git-dir "$gitdir" "$url" "$sm_path" ||
> +		(
> +			clear_local_git_env
> +			git clone $quiet -n ${reference:+"$reference"} \
> +				--separate-git-dir "$gitdir" "$url" "$sm_path"
> +		) ||
>  		die "$(eval_gettext "Clone of '\$url' into submodule path
> '\$sm_path' failed")"

Line-wrapped broken patch?

>  	fi
>
> diff --git a/t/t7409-submodule-detached-worktree.sh
> b/t/t7409-submodule-detached-worktree.sh
> new file mode 100755
> index 0000000..db75642
> --- /dev/null
> +++ b/t/t7409-submodule-detached-worktree.sh
> @@ -0,0 +1,61 @@
> +#!/bin/sh
> +#
> +# Copyright (c) 2012 Daniel Graña
> +#
> +
> +test_description='Test submodules on detached working tree
> +
> +This test verifies that "git submodule" initialization, update and
> addition works

Line-wrapped broken patch?

> +on detahced working trees
> +'
> +
> +TEST_NO_CREATE_REPO=1
> +. ./test-lib.sh
> +
> +test_expect_success 'submodule on detached working tree' '
> +	git init --bare remote &&
> +	test_create_repo bundle1 &&
> +	(cd bundle1 && test_commit "shoot") &&
> +	mkdir home &&
> +	(
> +		cd home &&
> +		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
> +		git clone --bare ../remote .dotfiles &&
> +		git submodule add ../bundle1 .vim/bundle/sogood &&
> +		test_commit "sogood" &&
> +		git push origin master
> +	) &&

Don't you want to verify not just commands succeed, but leave a
reasonable result, e.g. by running "git rev-parse HEAD" in "remote"
and comparing the output with "git rev-parse HEAD" in .dotfiles, or
something?

> +	mkdir home2 &&
> +	(
> +		cd home2 &&
> +		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
> +		git clone --bare ../remote .dotfiles &&
> +		git submodule update --init

Likewise.  What state, other than "submodule update --init" does not
barf, do you expect to see?

> +	)
> +'
> +
> +test_expect_success 'submodule on detached working pointed by core.worktree' '
> +	mkdir home3 &&
> +	(
> +		cd home3 &&
> +		export GIT_DIR="$(pwd)/.dotfiles" &&
> +		git clone --bare ../remote "$GIT_DIR" &&
> +		git config core.bare false &&
> +		git config core.worktree .. &&
> +		git submodule add ../bundle1 .vim/bundle/dupe &&
> +		test_commit "dupe" &&
> +		git push origin master

Likewise.

> +	) &&
> +	(
> +		cd home &&
> +		export GIT_DIR="$(pwd)/.dotfiles" &&
> +		git config core.bare false &&
> +		git config core.worktree .. &&
> +		git pull &&
> +		git submodule update &&
> +		git submodule status &&
> +		test -d .vim/bundle/dupe

Likewise.  Is there something you want to verify in the branches in
the submodule and its working tree, other than the existence of that
directory?

> +	)
> +'
> +
> +test_done

Re: [PATCH] Solve git-submodule issues with detached work trees

[Jens Lehmann]

Am 23.07.2012 07:09, schrieb Junio C Hamano:
> Daniel Graña <dangra@gmail.com> writes:
> 
>> A common way to track dotfiles with git is using GIT_DIR and
>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>
>>     git init --bare ~/.dotfiles
>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>
>>     dotfiles add ~/.bashrc
>>     dotfiles commit -a -m "add my bashrc"
>>     ...
>>
>> but git-submodule complains when trying to add submodules:
>>
>>     dotfiles submodule add http://path.to/submodule
>>     fatal: working tree '/home/user' already exists.
>>
>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>> working tree.
>>
>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>> ---
> 
> I think this is in line with what we discussed earlier on list when
> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
> the last time.  Jens?

Yes, I think this is the only way submodules in current git can
be used with the GIT_DIR and GIT_WORK_TREE environment variables:
set them when adding or initializing the submodule and always use
the same settings when accessing them later. Daniel's dotfile
alias achieves exactly that, so his fix looks good. But I agree
the tests should be improved as you already pointed out.

Re: [PATCH] Solve git-submodule issues with detached work trees

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> Am 23.07.2012 07:09, schrieb Junio C Hamano:
>> Daniel Graña <dangra@gmail.com> writes:
>> 
>>> A common way to track dotfiles with git is using GIT_DIR and
>>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>>
>>>     git init --bare ~/.dotfiles
>>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>>
>>>     dotfiles add ~/.bashrc
>>>     dotfiles commit -a -m "add my bashrc"
>>>     ...
>>>
>>> but git-submodule complains when trying to add submodules:
>>>
>>>     dotfiles submodule add http://path.to/submodule
>>>     fatal: working tree '/home/user' already exists.
>>>
>>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>>> working tree.
>>>
>>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>>> ---
>> 
>> I think this is in line with what we discussed earlier on list when
>> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
>> the last time.  Jens?
>
> Yes, I think this is the only way submodules in current git can
> be used with the GIT_DIR and GIT_WORK_TREE environment variables:
> set them when adding or initializing the submodule and always use
> the same settings when accessing them later. Daniel's dotfile
> alias achieves exactly that, so his fix looks good. But I agree
> the tests should be improved as you already pointed out.

Thanks for a quick review.  The "the only way ... in current git can
be used" part makes it sound as if it is a somewhat suboptimal ugly
workaround, but if that is what you meant, what is a more optimal
and less ugly way that you have in mind?

If you want to move .git out of way with GIT_DIR and if you want to
sit somewhere different from the top of your working tree, you must
use GIT_WORK_TREE (or core.worktree) to tell where the top resides,
whether your project use submodules or not, so I do not think it is
an ugly workaround but is the one true way to use such a dismembered
layout, I would think.

Re: [PATCH] Solve git-submodule issues with detached work trees

[Daniel Graña]

On Mon, Jul 23, 2012 at 2:38 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 23.07.2012 07:09, schrieb Junio C Hamano:
>> Daniel Graña <dangra@gmail.com> writes:
>>
>>> A common way to track dotfiles with git is using GIT_DIR and
>>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>>
>>>     git init --bare ~/.dotfiles
>>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>>
>>>     dotfiles add ~/.bashrc
>>>     dotfiles commit -a -m "add my bashrc"
>>>     ...
>>>
>>> but git-submodule complains when trying to add submodules:
>>>
>>>     dotfiles submodule add http://path.to/submodule
>>>     fatal: working tree '/home/user' already exists.
>>>
>>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>>> working tree.
>>>
>>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>>> ---
>>
>> I think this is in line with what we discussed earlier on list when
>> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
>> the last time.  Jens?
>
> Yes, I think this is the only way submodules in current git can
> be used with the GIT_DIR and GIT_WORK_TREE environment variables:
> set them when adding or initializing the submodule and always use
> the same settings when accessing them later. Daniel's dotfile
> alias achieves exactly that, so his fix looks good. But I agree
> the tests should be improved as you already pointed out.

Hi Jens, Junio.

Thanks for reviewing the patch, glad to hear this is acceptable way to
fix my itch.

I will work soon on getting the test cases improved and the patch
submitted without line wrapping.

thanks again,
Daniel.

Re: [PATCH] Solve git-submodule issues with detached work trees

[Richard Hartmann]

On Mon, Jul 23, 2012 at 7:09 AM, Junio C Hamano <gitster@pobox.com> wrote:

> I think this is in line with what we discussed earlier on list when
> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
> the last time.  Jens?

Yes, it is.

I still have your email marked as TODO to get back to you as I am
still unsure how to solve this whole mess in a way that works with
vcsh[1].


Richard

PS: Thanks for being so dedicated you all. I really do appreciate this
more than you most likely suspect.

[1] vcsh

Re: [PATCH] Solve git-submodule issues with detached work trees

[Richard Hartmann]

PPS: Yes, I know that I am replying in a patch thread. I will try it asap.

Re: [PATCH] Solve git-submodule issues with detached work trees

[Jens Lehmann]

Am 23.07.2012 20:21, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> Am 23.07.2012 07:09, schrieb Junio C Hamano:
>>> Daniel Graña <dangra@gmail.com> writes:
>>>
>>>> A common way to track dotfiles with git is using GIT_DIR and
>>>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>>>
>>>>     git init --bare ~/.dotfiles
>>>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>>>
>>>>     dotfiles add ~/.bashrc
>>>>     dotfiles commit -a -m "add my bashrc"
>>>>     ...
>>>>
>>>> but git-submodule complains when trying to add submodules:
>>>>
>>>>     dotfiles submodule add http://path.to/submodule
>>>>     fatal: working tree '/home/user' already exists.
>>>>
>>>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>>>> working tree.
>>>>
>>>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>>>> ---
>>>
>>> I think this is in line with what we discussed earlier on list when
>>> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
>>> the last time.  Jens?
>>
>> Yes, I think this is the only way submodules in current git can
>> be used with the GIT_DIR and GIT_WORK_TREE environment variables:
>> set them when adding or initializing the submodule and always use
>> the same settings when accessing them later. Daniel's dotfile
>> alias achieves exactly that, so his fix looks good. But I agree
>> the tests should be improved as you already pointed out.
> 
> Thanks for a quick review.  The "the only way ... in current git can
> be used" part makes it sound as if it is a somewhat suboptimal ugly
> workaround, but if that is what you meant, what is a more optimal
> and less ugly way that you have in mind?

I don't see it as an ugly workaround, I just don't see a way to get
the flexibility GIT_DIR and GIT_WORK_TREE offer for repos without
submodules (e.g. put the work tree temporarily somewhere else for
deployment or compare the current work tree with the history in a
different git directory) without changing core git. I have no idea
how many people use this flexibility but when submodules are
involved, you have to stick to the choices you made in the first
place or things will break. Right now the two relative links tie
exactly two directories together as git dir and work tree. I suspect
e.g. some setups using symlinks could benefit if we could get rid
of one of them or even both.

We could get rid of the core.worktree setting by assuming that the
directory a gitfile was found in is the root of the repo's work
tree (unless configured otherwise). And we could introduce a new
gitfile notation which gives the git dir relative to a parent repo's
git dir (e.g. "parent: modules/name" or such). But I'm not sure if
it is worth the hassle.

> If you want to move .git out of way with GIT_DIR and if you want to
> sit somewhere different from the top of your working tree, you must
> use GIT_WORK_TREE (or core.worktree) to tell where the top resides,
> whether your project use submodules or not, so I do not think it is
> an ugly workaround but is the one true way to use such a dismembered
> layout, I would think.

That is a perfect use case and we should make it work.

Re: [PATCH] Solve git-submodule issues with detached work trees

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> We could get rid of the core.worktree setting by assuming that the
> directory a gitfile was found in is the root of the repo's work
> tree (unless configured otherwise).

Now you lost me.  If you have .git that is not a directory but is a
gitfile, then you do not need GIT_DIR nor GIT_WORK_TREE in the first
place, no?

Re: [PATCH] Solve git-submodule issues with detached work trees

[Jens Lehmann]

Am 23.07.2012 22:34, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> We could get rid of the core.worktree setting by assuming that the
>> directory a gitfile was found in is the root of the repo's work
>> tree (unless configured otherwise).
> 
> Now you lost me.  If you have .git that is not a directory but is a
> gitfile, then you do not need GIT_DIR nor GIT_WORK_TREE in the first
> place, no?

Not inside the submodule, me thinks they only make sense in the
superproject (that's why we clean the environment before working
inside the submodule). But setting the superproject's GIT_WORK_TREE
to something else won't work for an already initialized submodule,
as the core.worktree setting will still point to the old work tree
which was set when the submodule was initialized. One could expect
the submodule's work tree to be $GIT_WORK_TREE/$sm_path when
GIT_WORK_TREE is set, but it isn't.

Re: [PATCH] Solve git-submodule issues with detached work trees

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> Not inside the submodule, me thinks they only make sense in the
> superproject (that's why we clean the environment before working
> inside the submodule).

Yes, that is fundamental and the only sane behaviour that comes from
what submodules are.  They are free-standing projects.  Not clearing
these environments when Git recurses into submodule was simply a bug
(the original bug was that we added recursion without thinking these
things through).

Hence...

> But setting the superproject's GIT_WORK_TREE
> to something else won't work for an already initialized submodule,

... if you point the _root_ of the toplevel superproject with
GIT_WORK_TREE (and the repository of the superproject with GIT_DIR),
and chdir into one of its submodule's working tree, we shouldn't say
$GIT_WORK_TREE/$smpath is the submodule world.  That will make it
impossible to work only on submodules by setting GIT_WORK_TREE to
point at its root level (and the submodule repository with GIT_DIR).