git submodule vs GIT_WORK_TREE

git submodule vs GIT_WORK_TREE

[Richard Hartmann]

Hi all,

vcsh [1] has had an open bug[2] filed against it for some time now. I
came to the conclusion that I can not properly work around this within
the scope of vcsh so I am hoping for help on this list.

As you are most likely aware, `git submodule` allows you to access a
separate git repository in a sub-directory, e.g. $submodule, while
$GIT_WORK_TREE forces git to operate in a specific directory. Combine
the two and the result is less than ideal. git is forced to operate in
$GIT_WORK_TREE while it should operate on $GIT_WORK_TREE/$submodule,
instead.

I think there are two options, here:

* Make git append $submodule to $GIT_WORK_TREE, i.e. operate on
$GIT_WORK_TREE/$submodule
* Introduce $GIT_WORK_TREE_RELATIVE which allows submodule and
potentially other commands to append to it


Did I miss another option? Would anyone be willing to fix/implement this?


Thanks,
Richard

[1] https://github.com/RichiH/vcsh
[2] https://github.com/RichiH/vcsh/issues/27

Re: git submodule vs GIT_WORK_TREE

[Junio C Hamano]

Richard Hartmann <richih.mailinglist@gmail.com> writes:

> As you are most likely aware, `git submodule` allows you to access a
> separate git repository in a sub-directory, e.g. $submodule, while
> $GIT_WORK_TREE forces git to operate in a specific directory. Combine
> the two and the result is less than ideal. git is forced to operate in
> $GIT_WORK_TREE while it should operate on $GIT_WORK_TREE/$submodule,
> instead.

The end user should not be asked to anything special.

When the user exports GIT_WORK_TREE to tell git that the root of the
working tree the user wants to work on resides there (which is
needed when the user exports GIT_DIR to tell git that the user wants
to work on that repository), that wish obviously applies only to
that repository.  If git decides to visit the working tree of a
different repository (e.g. a checkout of a submodule bound to the
project GIT_WORK_TREE points at), even if it is done in response to
the user action (e.g. like passing "--recurse-submodules" option),
it should adjust GIT_WORK_TREE and GIT_DIR to be appropriate for
operations in the submodule repository while doing so.  If the more
recent "recursive" behaviour forgets to do so, it simply is a bug.

Re: git submodule vs GIT_WORK_TREE

[Jens Lehmann]

Am 26.06.2012 18:07, schrieb Junio C Hamano:
> Richard Hartmann <richih.mailinglist@gmail.com> writes:
> 
>> As you are most likely aware, `git submodule` allows you to access a
>> separate git repository in a sub-directory, e.g. $submodule, while
>> $GIT_WORK_TREE forces git to operate in a specific directory. Combine
>> the two and the result is less than ideal. git is forced to operate in
>> $GIT_WORK_TREE while it should operate on $GIT_WORK_TREE/$submodule,
>> instead.

Richard, could you please tell us what git commands behave strange
in what way and what your expectations were? A short test shows e.g.
"git submodule foreach" operates on $(pwd)/submodule even though
GIT_WORK_TREE is set while at the same time git status shows the same
submodule as missing because it is looking for it in $GIT_WORK_TREE.
Is that what you are talking about?

> The end user should not be asked to anything special.
> 
> When the user exports GIT_WORK_TREE to tell git that the root of the
> working tree the user wants to work on resides there (which is
> needed when the user exports GIT_DIR to tell git that the user wants
> to work on that repository), that wish obviously applies only to
> that repository.  If git decides to visit the working tree of a
> different repository (e.g. a checkout of a submodule bound to the
> project GIT_WORK_TREE points at), even if it is done in response to
> the user action (e.g. like passing "--recurse-submodules" option),
> it should adjust GIT_WORK_TREE and GIT_DIR to be appropriate for
> operations in the submodule repository while doing so.  If the more
> recent "recursive" behaviour forgets to do so, it simply is a bug.

I'm not sure what you mean by "appropriate for operations in the
submodule repository". Should the submodule work tree be searched
for under $GIT_WORK_TREE of the superproject or under $(pwd)?

As far as I can see all submodule code consistently clears all
environment variables used by git before descending into a
submodule (at least since February 2010 and 5ce9086dd). Maybe we
should change that so it sets the GIT_WORK_TREE environment to
"$GIT_WORK_TREE/submodule" to be consistent?

Re: git submodule vs GIT_WORK_TREE

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> Am 26.06.2012 18:07, schrieb Junio C Hamano:
> ...
>> When the user exports GIT_WORK_TREE to tell git that the root of the
>> working tree the user wants to work on resides there (which is
>> needed when the user exports GIT_DIR to tell git that the user wants
>> to work on that repository), that wish obviously applies only to
>> that repository.  If git decides to visit the working tree of a
>> different repository (e.g. a checkout of a submodule bound to the
>> project GIT_WORK_TREE points at), even if it is done in response to
>> the user action (e.g. like passing "--recurse-submodules" option),
>> it should adjust GIT_WORK_TREE and GIT_DIR to be appropriate for
>> operations in the submodule repository while doing so.  If the more
>> recent "recursive" behaviour forgets to do so, it simply is a bug.
>
> I'm not sure what you mean by "appropriate for operations in the
> submodule repository". Should the submodule work tree be searched
> for under $GIT_WORK_TREE of the superproject or under $(pwd)?

I think either

 (1) unset GIT_WORK_TREE/GIT_DIR if the process chdirs to
     $GIT_WORK_TREE/submodule and $GIT_WORK_TREE/submodule/.git is
     the controlling reopsitory of that submodule working tree, or

 (2) set GIT_WORK_TREE/GIT_DIR to point at the working tree and
     repository of the submodule.

would be appropriate.

> As far as I can see all submodule code consistently clears all
> environment variables used by git before descending into a
> submodule (at least since February 2010 and 5ce9086dd). Maybe we
> should change that so it sets the GIT_WORK_TREE environment to
> "$GIT_WORK_TREE/submodule" to be consistent?

If the user has to use GIT_WORK_TREE to mark the root level of the
superproject working tree as such, it is very likely that the
controlling repository of the superproject does not live in the
$GIT_WORK_TREE/.git directory (in other words, $GIT_DIR points at
somewhere else).  Exporting GIT_WORK_TREE/submodule as the new value
of GIT_WORK_TREE is sensible, but I do not see a reasonable way to
deduce the value of GIT_DIR for the submodule in such a case.  The
controlling repository of the superproject is located somewhere
random; there is no reason to assume the repository for the
submodule is somewhere at fixed relation to it.

Does it mean the short answer to Richard's situation is "Don't do
it"?  I am not sure, but it is starting to sound like it.

Re: git submodule vs GIT_WORK_TREE

[Jens Lehmann]

Am 26.06.2012 21:51, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> Am 26.06.2012 18:07, schrieb Junio C Hamano:
>> ...
>>> When the user exports GIT_WORK_TREE to tell git that the root of the
>>> working tree the user wants to work on resides there (which is
>>> needed when the user exports GIT_DIR to tell git that the user wants
>>> to work on that repository), that wish obviously applies only to
>>> that repository.  If git decides to visit the working tree of a
>>> different repository (e.g. a checkout of a submodule bound to the
>>> project GIT_WORK_TREE points at), even if it is done in response to
>>> the user action (e.g. like passing "--recurse-submodules" option),
>>> it should adjust GIT_WORK_TREE and GIT_DIR to be appropriate for
>>> operations in the submodule repository while doing so.  If the more
>>> recent "recursive" behaviour forgets to do so, it simply is a bug.
>>
>> I'm not sure what you mean by "appropriate for operations in the
>> submodule repository". Should the submodule work tree be searched
>> for under $GIT_WORK_TREE of the superproject or under $(pwd)?
> 
> I think either
> 
>  (1) unset GIT_WORK_TREE/GIT_DIR if the process chdirs to
>      $GIT_WORK_TREE/submodule and $GIT_WORK_TREE/submodule/.git is
>      the controlling reopsitory of that submodule working tree, or
> 
>  (2) set GIT_WORK_TREE/GIT_DIR to point at the working tree and
>      repository of the submodule.
> 
> would be appropriate.

Agreed.

>> As far as I can see all submodule code consistently clears all
>> environment variables used by git before descending into a
>> submodule (at least since February 2010 and 5ce9086dd). Maybe we
>> should change that so it sets the GIT_WORK_TREE environment to
>> "$GIT_WORK_TREE/submodule" to be consistent?
> 
> If the user has to use GIT_WORK_TREE to mark the root level of the
> superproject working tree as such, it is very likely that the
> controlling repository of the superproject does not live in the
> $GIT_WORK_TREE/.git directory (in other words, $GIT_DIR points at
> somewhere else).  Exporting GIT_WORK_TREE/submodule as the new value
> of GIT_WORK_TREE is sensible, but I do not see a reasonable way to
> deduce the value of GIT_DIR for the submodule in such a case.  The
> controlling repository of the superproject is located somewhere
> random; there is no reason to assume the repository for the
> submodule is somewhere at fixed relation to it.
> 
> Does it mean the short answer to Richard's situation is "Don't do
> it"?  I am not sure, but it is starting to sound like it.

Not at all, I was just trying to reach consensus on what the user
can reasonably expect when setting GIT_WORK_TREE in the presence of
submodules. I will look into it to see how we can handle the cases
where GIT_WORK_TREE and/or GIT_DIR are set.

Re: git submodule vs GIT_WORK_TREE

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> Am 26.06.2012 21:51, schrieb Junio C Hamano:
> ...
>> If the user has to use GIT_WORK_TREE to mark the root level of the
>> superproject working tree as such, it is very likely that the
>> controlling repository of the superproject does not live in the
>> $GIT_WORK_TREE/.git directory (in other words, $GIT_DIR points at
>> somewhere else).  Exporting GIT_WORK_TREE/submodule as the new value
>> of GIT_WORK_TREE is sensible, but I do not see a reasonable way to
>> deduce the value of GIT_DIR for the submodule in such a case.  The
>> controlling repository of the superproject is located somewhere
>> random; there is no reason to assume the repository for the
>> submodule is somewhere at fixed relation to it.
>> 
>> Does it mean the short answer to Richard's situation is "Don't do
>> it"?  I am not sure, but it is starting to sound like it.
>
> Not at all, I was just trying to reach consensus on what the user
> can reasonably expect when setting GIT_WORK_TREE in the presence of
> submodules. I will look into it to see how we can handle the cases
> where GIT_WORK_TREE and/or GIT_DIR are set.

Thanks, then I'll let you look into it.  I do not see how you could
determine GIT_DIR for the submodule sensibly if the superproject's
working tree does not have its GIT_DIR as ".git" at the top-level,
though.

Declaring that a checked out submodule _must_ have its controlling
repository as ".git" at its root level, i.e. it should be accessible
without using GIT_WORK_TREE/GIT_DIR at all just like a normal Git
repository is, would be a clean way to avoid the issue altogether
and it still will allow the top-level superproject to be structured
in a funny way to require GIT_WORK_TREE/GIT_DIR to access it, but I
am not sure if that is too restrictive for people who for some
unknown reason want to use these environment variables to create
repositories with a strange layout.

Re: git submodule vs GIT_WORK_TREE

[Richard Hartmann]

Mangling the email I respond to out of order.

On Wed, Jun 27, 2012 at 12:53 AM, Junio C Hamano <gitster@pobox.com> wrote:

> but I
> am not sure if that is too restrictive for people who for some
> unknown reason want to use these environment variables to create
> repositories with a strange layout.

This is exactly the point. For vcsh[1] to work, I need to set it up like this:

    richih@titanium  ~ % vcsh enter zsh
    richih@titanium [zsh] [2] (git)-[master] ~ % echo $GIT_WORK_TREE
    /home/richih
    richih@titanium [zsh] [2] (git)-[master] ~ % echo $GIT_DIR
    /home/richih/.config/vcsh/repo.d/zsh.git
    richih@titanium [zsh] [2] (git)-[master] ~ %


> Declaring that a checked out submodule _must_ have its controlling
> repository as ".git" at its root level, i.e. it should be accessible
> without using GIT_WORK_TREE/GIT_DIR at all just like a normal Git
> repository is, would be a clean way to avoid the issue altogether
> and it still will allow the top-level superproject to be structured
> in a funny way to require GIT_WORK_TREE/GIT_DIR to access it,

I fear this would make vcsh unable to support submodules, ever.



Richard

[1] https://github.com/RichiH/vcsh

Re: git submodule vs GIT_WORK_TREE

[Junio C Hamano]

Richard Hartmann <richih.mailinglist@gmail.com> writes:

> Mangling the email I respond to out of order.
>
> On Wed, Jun 27, 2012 at 12:53 AM, Junio C Hamano <gitster@pobox.com> wrote:
>
>> but I
>> am not sure if that is too restrictive for people who for some
>> unknown reason want to use these environment variables to create
>> repositories with a strange layout.
>
> This is exactly the point. For vcsh[1] to work, I need to set it up like this:
>
>     richih@titanium  ~ % vcsh enter zsh
>     richih@titanium [zsh] [2] (git)-[master] ~ % echo $GIT_WORK_TREE
>     /home/richih
>     richih@titanium [zsh] [2] (git)-[master] ~ % echo $GIT_DIR
>     /home/richih/.config/vcsh/repo.d/zsh.git
>     richih@titanium [zsh] [2] (git)-[master] ~ %
>
>
>> Declaring that a checked out submodule _must_ have its controlling
>> repository as ".git" at its root level, i.e. it should be accessible
>> without using GIT_WORK_TREE/GIT_DIR at all just like a normal Git
>> repository is, would be a clean way to avoid the issue altogether
>> and it still will allow the top-level superproject to be structured
>> in a funny way to require GIT_WORK_TREE/GIT_DIR to access it,
>
> I fear this would make vcsh unable to support submodules, ever.

Then think of a workable alternative.  As it stands, I would have to
say that what vcsh tries to do with git is broken with respect to
submodules.

Re: git submodule vs GIT_WORK_TREE

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>>> Declaring that a checked out submodule _must_ have its controlling
>>> repository as ".git" at its root level, i.e. it should be accessible
>>> without using GIT_WORK_TREE/GIT_DIR at all just like a normal Git
>>> repository is, would be a clean way to avoid the issue altogether
>>> and it still will allow the top-level superproject to be structured
>>> in a funny way to require GIT_WORK_TREE/GIT_DIR to access it,
>>
>> I fear this would make vcsh unable to support submodules, ever.
>
> Then think of a workable alternative.  As it stands, I would have to
> say that what vcsh tries to do with git is broken with respect to
> submodules.

Just to avoid misunderstanding.  I didn't mean "vcsh and vcsh alone
is broken while other solutions work just fine".  What I meant was
"I agree with you that what vcsh tries to do is unworkable".

Re: git submodule vs GIT_WORK_TREE

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>>> Declaring that a checked out submodule _must_ have its controlling
>>> repository as ".git" at its root level, i.e. it should be accessible
>>> without using GIT_WORK_TREE/GIT_DIR at all just like a normal Git
>>> repository is, would be a clean way to avoid the issue altogether
>>> and it still will allow the top-level superproject to be structured
>>> in a funny way to require GIT_WORK_TREE/GIT_DIR to access it,

Let's think aloud a bit with a simple example.  Imagine there is a
directory hierarchy $HOME/a/b with a file $HOME/a/b/c in it.

'$HOME/a' is a working tree of a Git managed project.  In a normal
layout, there is a $HOME/a/.git directory there that houses its Git
repository, and everything (recursively) outside $HOME/a/.git in the
directory $HOME/a are working tree files of that repository.  You
can

	cd $HOME/a
        git add b/c

to add the file at path b/c, and commit the changes to it.

You can relocate $HOME/a/.git elsewhere, say $HOME/git/a.git, by
exporting GIT_DIR=$HOME/git/a.git as long as you work at the top
level of the working tree.  When GIT_DIR alone is set this way, you
are giving no clue to Git where the top-level of the working tree is
(remember, you relocated $HOME/a/.git away so there is no ".git" in
the $HOME/a directory), so your $(cwd) will be always taken as the
top level of the working tree.

It is inconvenient if you want to work in $HOME/a/b.  So you can do
this:

	GIT_WORK_TREE=$HOME/a
        export GIT_WORK_TREE
        cd $HOME/a/b
	git add c

and everything works again.

Or $HOME/a/b may be a separate project, and $HOME/a/.git repository
binds that as a submodule.  In a normal layout, a $HOME/a/b/.git
directory is there that houses its Git repository.  'c' is a file
that appears at the top level of that working tree.  When you are at
the top-level of the superproject's working tree, you see b/ and Git
knows it is a submodule, not a subdirectory of the superproject.  It
cannot allow "git add b/c" to the superproject.

And this will continue to work with GIT_DIR/GIT_WORK_TREE when you
move $HOME/a/.git elsewhere.

What happens if you also moved $HOME/a/b/.git elsewhere?

This will still work when you want to work inside the submodule:

	GIT_WORK_TREE=$HOME/a/b
        export GIT_WORK_TREE
        cd $HOME/a/b
	git add c

to record the change to 'c' at the top-level of the working tree of
the submodule.

*BUT*

There is no ".git" in $HOME/a anywhere, so this breaks when you are
in the superproject's working tree. There is no longer anything that
tells Git that the $HOME/a/b path is a project boundary.

You could look at the index to notice b is known as a submodule, but
that is not a usable solution in general.  You could have done "git
rm --cached" it, perhaps because you wanted to do something clever
with it, e.g. in preparation to move it, in which case we still
would want to treat it as the working tree of a separate Git
repository that is not yet (or no longer) connected to the working
tree of the $HOME/a/ repository.

By the way, without any of the GIT_DIR/GIT_WORK_TREE hack, the
following sequence does not work correctly (it is a bug nobody cares
about), and we may want to correct it.

	mkdir -p /var/tmp/x/a/b
        cd /var/tmp/x/a
        git init ;# create a/.git
        git init ;# create a/b/.git
        >c && git add c ;# a/b/.git knows about c at the toplevel
        cd .. ;# back at a
        git add b/c

Even though a/b/.git does not yet have a commit and 'b' has not been
registered as a submodule to its index, Git run in 'a' should notice
that b/c belongs to somebody else, and refuse to grab it.  It has
enough cue (namely, the presense of a/b/.git and the directory being
a proper Git repository) to do so, but it currently does not use
that clue.

Re: git submodule vs GIT_WORK_TREE

[Richard Hartmann]

First of all, thanks for your detailed analysis; it cleared things up
for me quite a bit.

I am not sure if this would work in all cases, but let's assume there
is a new GIT_WORK_ROOT which will always point to the top level of a
given repository (it would stay the same for submodules, as well).

As for storage of .git, GIT_DIR_ROOT would always point to a directory
where git data is stored and GIT_DIR_NAME to the relative path within
GIT_DIR_ROOT. Admittedly, this may be rather specific to vcsh, but I
don't see any other (reliable) option to support submodules.


Thanks,
Richard

Re: git submodule vs GIT_WORK_TREE

[Junio C Hamano]

Richard Hartmann <richih.mailinglist@gmail.com> writes:

> First of all, thanks for your detailed analysis; it cleared things up
> for me quite a bit.
>
> I am not sure if this would work in all cases, but let's assume there
> is a new GIT_WORK_ROOT which will always point to the top level of a
> given repository (it would stay the same for submodules, as well).

I do not think such a variable is necessary, and I do not think it
helps anything either.

As a submodule is an independent project and does not care if it is
embedded inside another project (i.e. superproject), when you are
working in $HOME/a/b repository (going back to the example in the
previous message), whether using b/.git or GIT_WORK_TREE=$HOME/a/b
as the clue to mark the root of the working tree of the project you
are working in, there is no reason for Git to figure out that there
is a containing superproject at $HOME/a, let alone that its
controlling repository is at $HOME/a/.git (or somewhere else).

The problem is going the other way around.

When you are working in the superproject, i.e. $HOME/a, there is a
need to know when you cross the project boundary at $HOME/a/b (and
if you know $HOME/a/b is the project boundary, that automatically
means $HOME/a/b is the root level of the working tree of the other
project, so Git with "--recursive" option _ought_ to be aboe to
export GIT_WORK_TREE one it goes into the working tree of the
submodule without anything else).

But even if you can know where the project boundary is in the
working tree space, that does not automatically mean you will be
able to know where the controlling repository for $HOME/a/b is, if
you moved it away from its natural location $HOME/a/b/.git to
somewhere else.  There needs to be a way to tell Git about that.

And an obvious way to do so is to have $HOME/a/b/.git; it can be a
real repository, which would make things simple, but it does not
have to be.  It can be the "gitfile:" thing that points at somewhere
else.  In recent versions of git, submodule working trees have this
as a "gitfile:" that points at $GIT_DIR/modules/$name of the
superproject, so that the working tree can be removed when checking
out an old version of superproject before such a submodule was added
to the project without losing the controlling repository.

So I do not see a need for either WORK_ROOT or DIR_ROOT at all.

The superproject (e.g. $HOME/a) may have its controlling repository
elsewhere (you are naming its location with $GIT_DIR when you name
the working tree of it with $GIT_WORK_TREE), and that controlling
repository can have modules/b directory that is the controlling
repository for the submodule whose working tree is at $HOME/a/b;
there is no need for anything else as long as there is enough clue
for the system to find out that $HOME/a/b is the project boundary.

Re: git submodule vs GIT_WORK_TREE

[Richard Hartmann]

On Tue, Jun 26, 2012 at 8:27 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:

> Richard, could you please tell us what git commands behave strange
> in what way and what your expectations were?

A real-life example can be found at:

  https://gist.github.com/2992239

lines 354 in the second and 353 in the third paste block.


> As far as I can see all submodule code consistently clears all
> environment variables used by git before descending into a
> submodule (at least since February 2010 and 5ce9086dd). Maybe we
> should change that so it sets the GIT_WORK_TREE environment to
> "$GIT_WORK_TREE/submodule" to be consistent?

>From how I understand the problem, this would solve it.

-- 
Richard