Submodule relative URL problems

Submodule relative URL problems

[Lianheng Tong]

Hi All,

I am currently working on a project, call it A, which contains several sub-projects B, C, D.  The B, C, D are actually experimental input and outputs for simulation runs, which I need to keep track of the daily changes.  Most of these calculations are done on workstation, but I do need to occasionally clone the repositories to other machines, and then push the updates back.

I version track B, C and D with independent git repositories, and organisationally it makes sense for them to be subdirectories of A.  I would like to version control the overall project A, with B, C, D as submodules.

This is easily done on the main workstation (I will call it W1), with:

cd A
git init
for ii in B C D ; do
  git submodule ./${ii}/.git
done
git add .
git commit

Repository A/.git on W1 does not have an origin.  All seems to work fine with this step up on W1.

HOWEVER, problem starts to appear when I attempted to clone the overall repository A to another workstation W2:

git clone W1:<path to A on W1>/.git  <path to A on W2>
cd <path to A on W2>
git submodule init
git submodule update

git then produces an error indicating that:

W1:<path to A on W1>/.git/B/.git

is not a valid repository (i.e. directory not found)

The extra .git has been inserted in the URL to submodule repository B, C, and D.

This can be solved by changing the .gitmodules file, replacing ./B/.git to ../B/.git,  and then git submodule sync on W2.

However, this is far from being an optimal solution, because if .gitmodules on W1 is updated accordingly, then the submodule repository for B would point to

<path to A on W1>/../B/.git

which will be incorrect.  I could in principle define the URL for the submodule repositories as absolute paths, but keeping them relative to A has many advantages, such as I will be able to move A around, without effecting the main repositories on W1 in anyway, and I can rsync them to another disk for backup etc.

I would be very grateful if someone can offer me an solution to my problem.

Many thanks in advance!

Lianheng

================================================
Lianheng Tong                                    Tel: +44 79 1758 3822  
Room S4.02, Strand Building           Fax: +44 20 7848 2420  
Department of Physics                       lianheng.tong@kcl.ac.uk
Kings College London                                  
Strand, London WC2R 2LS, U.K.                         
================================================

Re: Submodule relative URL problems

[Jonathan Nieder]

Hi,

Lianheng Tong wrote:

> git clone W1:<path to A on W1>/.git  <path to A on W2>

Interesting.

Thoughts:

 * More typical usage is to clone from a bare repository (A.git), which
   wouldn't have this problem.  But I think your case is worth
   supporting, too.

 * What would you think of putting symlinks in A's .git directory?

	cd A/.git
	ln -s ../B ../C ../D .

 * Perhaps as a special case when the superproject is foo/.git, git
   should treat relative submodule paths as relative to foo/ instead
   of relative to foo/.git/.  I think that would take care of your
   case without breaking existing normal practices, though after the
   patch is made it still wouldn't take care of people using old
   versions of git without that patch.  What do you think?

Thanks,
Jonathan

Re: Re: Submodule relative URL problems

[Heiko Voigt]

Hi,

On Mon, Jan 13, 2014 at 11:55:18AM -0800, Jonathan Nieder wrote:
> Lianheng Tong wrote:
> 
> > git clone W1:<path to A on W1>/.git  <path to A on W2>
> 
> Interesting.
> 
> Thoughts:
> 
>  * More typical usage is to clone from a bare repository (A.git), which
>    wouldn't have this problem.  But I think your case is worth
>    supporting, too.
> 
>  * What would you think of putting symlinks in A's .git directory?
> 
> 	cd A/.git
> 	ln -s ../B ../C ../D .
> 
>  * Perhaps as a special case when the superproject is foo/.git, git
>    should treat relative submodule paths as relative to foo/ instead
>    of relative to foo/.git/.  I think that would take care of your
>    case without breaking existing normal practices, though after the
>    patch is made it still wouldn't take care of people using old
>    versions of git without that patch.  What do you think?

I do not fully get the repository layout, since some commands simply do
not work. Nevertheless I think what Lianheng Tong is running into is
the following:

 * If a superproject has *no remote* a relative submodule url is relative
   to the *superproject itself*
 * If a superproject has *a remote* a relative submodule url is relative
   to the *superprojects remote*

The simplest solution is: Have central bare repositories for everything
so that every workstation has the same remote.

The second solution: Make sure both repositories have each other as a
remote. But then you run into a chicken/egg problem when setting the two
up.

The interpretation of relative urls was a design decision back when the
relative urls were introduced. I am quite sure it would produce a lot of
fallout if we change that.

If I get your usecase wrong it would be very helpful if you could
provide us with a working script that creates the repository setup your
are using.

Cheers Heiko

Re: Submodule relative URL problems

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

>  * More typical usage is to clone from a bare repository (A.git), which
>    wouldn't have this problem.  But I think your case is worth
>    supporting, too.

I think the relative URL among nested submodules was specifically
designed for hosting environments that have forest of bare
repositories to serve as publishing or meeting points.  I personally
do not know where that "worth supporting" comes from, but if the
change can be done without confusing the codepaths involved, I
wouldn't object too much ;-)

>  * Perhaps as a special case when the superproject is foo/.git, git
>    should treat relative submodule paths as relative to foo/ instead
>    of relative to foo/.git/.  I think that would take care of your
>    case without breaking existing normal practices, though after the
>    patch is made it still wouldn't take care of people using old
>    versions of git without that patch.  What do you think?

If we were to start adding special cases, it may also be sensible to
clean up the more normal case of using subdirectories of bare
repositories to represent nestedness (i.e. you can have a submodule
"logs.git", but not "logs").  We could reuse the $GIT_DIR/modules/$sm_name
convention somehow perhaps?

Any change to implement the special case you suggest likely has to
touch the same place as such a change does, as both require some
reorganization of the code that traverses the pathnames to find
related repositories.