Multiple working trees with GIT ?

Multiple working trees with GIT ?

[Willy Tarreau]

Hi all,

I'm having long thoughts about how to use GIT to manage a distro. One of
the aspects which comes very often is the notion of "variant" for a
packaging. For instance, the whole project could consist in a list of packages
with their branches, but this list may vary depending on the platform, the
medium, etc... I was searching how to propagate common changes withing variants
with the least hassle.

I figured out that having one file list per variant will be very annoying. In
another project, that's already what I have and frankly, applying the same
change to 10 files is counter-productive. Since the lists will often be the
sames except for a few entries, and since most updates will be relevant to
all variants, I thought branches will be my best friends.

But I would like to be able to always access file lists, without having to
constantly git-checkout <variant-X>.

Finally, I found a very convenient solution, but I don't know if there are
any risks using it. If we can conclude to a riskless usage (with/without
some adjustments), I can contribute a script to setup this environment.

The idea is to have multiple working trees in a subdirectory of the normal
one. Some would probably like them to be movable anywhere else, but let's
not complicate things first. But in order not to constantly have to pull/push
in every tree, I set up the .git repo of each working tree with many symlinks :

  project/
          .git/ (normal repo)
          ... master checked out there by default ...
          worktree/
                   variant_a/
                             .git/ (symlinks)
                   variant_b/
                             .git/ (symlinks)
                   variant_c/
                             .git/ (symlinks)


Each variant is set up like this :
    4096 Jan 24 08:44 .git
      19 Jan 24 08:27 .git/HEAD
      28 Jan 24 08:30 .git/logs
     419 Jan 24 08:44 .git/logs/HEAD
      26 Jan 24 08:31 .git/logs/refs -> ../../../../.git/logs/refs
      18 Jan 24 08:31 .git/refs -> ../../../.git/refs
      21 Jan 24 08:31 .git/objects -> ../../../.git/objects
      18 Jan 24 08:31 .git/info -> ../../../.git/info
      19 Jan 24 08:31 .git/hooks -> ../../../.git/hooks
      25 Jan 24 08:31 .git/description -> ../../../.git/description
      20 Jan 24 08:31 .git/config -> ../../../.git/config
     118 Jan 24 08:44 .git/index
      22 Jan 24 08:31 .git/branches -> ../../../.git/branches
      63 Jan 24 08:44 .git/FETCH_HEAD
      41 Jan 24 08:44 .git/ORIG_HEAD

In fact, I found that each directory which hosts a HEAD file needs to
remain a directory because of this head, but all other dirs can be
symlinks to original tree.

This works pretty well. I can simply cd worktree/variant_a and work on a
file, or pull master, or even git-cherry-pick from other branches (pretty
convenient for this usage). But I don't know what caveats I may encounter.

Maybe there are other solutions too. I see that we tend to replace symlinks
everywhere with ref files. We might as well (in a far future version) accept
a file for ".git" which would contain a path to the central repo and the
branch's head.

I'm open to comments. Please keep me CC, I'm not subscribed to the git list.

Thanks,
Willy

Re: Multiple working trees with GIT ?

[Julian Phillips]

On Thu, 24 Jan 2008, Willy Tarreau wrote:

> Hi all,
>
> I'm having long thoughts about how to use GIT to manage a distro. One of
> the aspects which comes very often is the notion of "variant" for a
> packaging. For instance, the whole project could consist in a list of packages
> with their branches, but this list may vary depending on the platform, the
> medium, etc... I was searching how to propagate common changes withing variants
> with the least hassle.
>
> I figured out that having one file list per variant will be very annoying. In
> another project, that's already what I have and frankly, applying the same
> change to 10 files is counter-productive. Since the lists will often be the
> sames except for a few entries, and since most updates will be relevant to
> all variants, I thought branches will be my best friends.
>
> But I would like to be able to always access file lists, without having to
> constantly git-checkout <variant-X>.
>
> Finally, I found a very convenient solution, but I don't know if there are
> any risks using it. If we can conclude to a riskless usage (with/without
> some adjustments), I can contribute a script to setup this environment.
>
> The idea is to have multiple working trees in a subdirectory of the normal
> one. Some would probably like them to be movable anywhere else, but let's
> not complicate things first. But in order not to constantly have to pull/push
> in every tree, I set up the .git repo of each working tree with many symlinks :
>
>  project/
>          .git/ (normal repo)
>          ... master checked out there by default ...
>          worktree/
>                   variant_a/
>                             .git/ (symlinks)
>                   variant_b/
>                             .git/ (symlinks)
>                   variant_c/
>                             .git/ (symlinks)
>
>
> Each variant is set up like this :
>    4096 Jan 24 08:44 .git
>      19 Jan 24 08:27 .git/HEAD
>      28 Jan 24 08:30 .git/logs
>     419 Jan 24 08:44 .git/logs/HEAD
>      26 Jan 24 08:31 .git/logs/refs -> ../../../../.git/logs/refs
>      18 Jan 24 08:31 .git/refs -> ../../../.git/refs
>      21 Jan 24 08:31 .git/objects -> ../../../.git/objects
>      18 Jan 24 08:31 .git/info -> ../../../.git/info
>      19 Jan 24 08:31 .git/hooks -> ../../../.git/hooks
>      25 Jan 24 08:31 .git/description -> ../../../.git/description
>      20 Jan 24 08:31 .git/config -> ../../../.git/config
>     118 Jan 24 08:44 .git/index
>      22 Jan 24 08:31 .git/branches -> ../../../.git/branches
>      63 Jan 24 08:44 .git/FETCH_HEAD
>      41 Jan 24 08:44 .git/ORIG_HEAD
>
> In fact, I found that each directory which hosts a HEAD file needs to
> remain a directory because of this head, but all other dirs can be
> symlinks to original tree.
>
> This works pretty well. I can simply cd worktree/variant_a and work on a
> file, or pull master, or even git-cherry-pick from other branches (pretty
> convenient for this usage). But I don't know what caveats I may encounter.

You might want to have a look at the git-new-workdir script in contrib, it 
does basically the same thing.  It's been there for about 10 months now. 
It was based on an email from Junio:

http://article.gmane.org/gmane.comp.version-control.git/41513/

However, there are some caveats about using this approach, basically about 
the fact that there is nothing stopping you from updating refs that are 
currently checked out in another directory and causing yourself all sorts 
of pain ... the topic has cropped up a couple of times on the list since 
the script was added.

> Maybe there are other solutions too. I see that we tend to replace symlinks
> everywhere with ref files. We might as well (in a far future version) accept
> a file for ".git" which would contain a path to the central repo and the
> branch's head.

There was a suggestion for something not too dissimilar even before the 
new-workdir script:

http://thread.gmane.org/gmane.comp.version-control.git/33755

AFAIK it hasn't actually been implemented.

-- 
Julian

  ---
May your SO always know when you need a hug.

Re: Multiple working trees with GIT ?

[Johannes Schindelin]

Hi,

On Thu, 24 Jan 2008, Julian Phillips wrote:

> You might want to have a look at the git-new-workdir script in contrib, 
> it does basically the same thing.  It's been there for about 10 months 
> now. It was based on an email from Junio:
> 
> http://article.gmane.org/gmane.comp.version-control.git/41513/

FWIW I have a patch to do something like that in "git branch" itself.

> However, there are some caveats about using this approach, basically 
> about the fact that there is nothing stopping you from updating refs 
> that are currently checked out in another directory and causing yourself 
> all sorts of pain ... the topic has cropped up a couple of times on the 
> list since the script was added.

I agree; maybe we should have a telltale file 
"refs/heads/<bla>.checkedout" which is heeded by "git checkout" and "git 
branch -d/-D", as well as update_ref() (should only update that ref when 
it HEAD points to it)?

Ciao,
Dscho

Re: Multiple working trees with GIT ?

[Willy Tarreau]

On Thu, Jan 24, 2008 at 11:04:42AM +0000, Johannes Schindelin wrote:
> Hi,
> 
> On Thu, 24 Jan 2008, Julian Phillips wrote:
> 
> > You might want to have a look at the git-new-workdir script in contrib, 
> > it does basically the same thing.  It's been there for about 10 months 
> > now. It was based on an email from Junio:
> > 
> > http://article.gmane.org/gmane.comp.version-control.git/41513/
> 
> FWIW I have a patch to do something like that in "git branch" itself.
>
> > However, there are some caveats about using this approach, basically 
> > about the fact that there is nothing stopping you from updating refs 
> > that are currently checked out in another directory and causing yourself 
> > all sorts of pain ... the topic has cropped up a couple of times on the 
> > list since the script was added.
> 
> I agree; maybe we should have a telltale file 
> "refs/heads/<bla>.checkedout" which is heeded by "git checkout" and "git 
> branch -d/-D", as well as update_ref() (should only update that ref when 
> it HEAD points to it)?

Why not generalize this into HEAD.$branch (thus limiting to one checkout
per branch) or HEAD.$checkoutdir ?

Best regards,
Willy

Re: Multiple working trees with GIT ?

[Johannes Schindelin]

Hi,

On Thu, 24 Jan 2008, Willy Tarreau wrote:

> On Thu, Jan 24, 2008 at 11:04:42AM +0000, Johannes Schindelin wrote:
> 
> > On Thu, 24 Jan 2008, Julian Phillips wrote:
> > 
> > > You might want to have a look at the git-new-workdir script in 
> > > contrib, it does basically the same thing.  It's been there for 
> > > about 10 months now. It was based on an email from Junio:
> > > 
> > > http://article.gmane.org/gmane.comp.version-control.git/41513/
> > 
> > FWIW I have a patch to do something like that in "git branch" itself.
> >
> > > However, there are some caveats about using this approach, basically 
> > > about the fact that there is nothing stopping you from updating refs 
> > > that are currently checked out in another directory and causing 
> > > yourself all sorts of pain ... the topic has cropped up a couple of 
> > > times on the list since the script was added.
> > 
> > I agree; maybe we should have a telltale file 
> > "refs/heads/<bla>.checkedout" which is heeded by "git checkout" and 
> > "git branch -d/-D", as well as update_ref() (should only update that 
> > ref when it HEAD points to it)?
> 
> Why not generalize this into HEAD.$branch (thus limiting to one checkout 
> per branch) or HEAD.$checkoutdir ?

Because multiple working trees for the same repository will always be a 
second-class citizen.  And I would rather not affect the common case too 
much.

Having a "lock" file which is heeded by just a few places which are 
supposed to update refs (thinking about it, just update_ref() should be 
enough), is at least a well-contained change.

Ciao,
Dscho

Re: Multiple working trees with GIT ?

[Willy Tarreau]

On Thu, Jan 24, 2008 at 01:38:45PM +0000, Johannes Schindelin wrote:
> Hi,
> 
> On Thu, 24 Jan 2008, Willy Tarreau wrote:
> 
> > On Thu, Jan 24, 2008 at 11:04:42AM +0000, Johannes Schindelin wrote:
> > 
> > > On Thu, 24 Jan 2008, Julian Phillips wrote:
> > > 
> > > > You might want to have a look at the git-new-workdir script in 
> > > > contrib, it does basically the same thing.  It's been there for 
> > > > about 10 months now. It was based on an email from Junio:
> > > > 
> > > > http://article.gmane.org/gmane.comp.version-control.git/41513/
> > > 
> > > FWIW I have a patch to do something like that in "git branch" itself.
> > >
> > > > However, there are some caveats about using this approach, basically 
> > > > about the fact that there is nothing stopping you from updating refs 
> > > > that are currently checked out in another directory and causing 
> > > > yourself all sorts of pain ... the topic has cropped up a couple of 
> > > > times on the list since the script was added.
> > > 
> > > I agree; maybe we should have a telltale file 
> > > "refs/heads/<bla>.checkedout" which is heeded by "git checkout" and 
> > > "git branch -d/-D", as well as update_ref() (should only update that 
> > > ref when it HEAD points to it)?
> > 
> > Why not generalize this into HEAD.$branch (thus limiting to one checkout 
> > per branch) or HEAD.$checkoutdir ?
> 
> Because multiple working trees for the same repository will always be a 
> second-class citizen.  And I would rather not affect the common case too 
> much.

OK.

> Having a "lock" file which is heeded by just a few places which are 
> supposed to update refs (thinking about it, just update_ref() should be 
> enough), is at least a well-contained change.

indeed, with the appropriate warnings/error messages, that makes a lot of sense.

Cheers,
Willy

Re: Multiple working trees with GIT ?

[Willy Tarreau]

Hi Julian,

On Thu, Jan 24, 2008 at 09:59:05AM +0000, Julian Phillips wrote:
(...)
> >This works pretty well. I can simply cd worktree/variant_a and work on a
> >file, or pull master, or even git-cherry-pick from other branches (pretty
> >convenient for this usage). But I don't know what caveats I may encounter.
> 
> You might want to have a look at the git-new-workdir script in contrib, it 
> does basically the same thing.  It's been there for about 10 months now. 
> It was based on an email from Junio:
> 
> http://article.gmane.org/gmane.comp.version-control.git/41513/

Interesting lecture, thanks for the pointer. At least now I know that it is
not too much exotic.

> However, there are some caveats about using this approach, basically about 
> the fact that there is nothing stopping you from updating refs that are 
> currently checked out in another directory and causing yourself all sorts 
> of pain ... the topic has cropped up a couple of times on the list since 
> the script was added.

hmmm good point. Given that I'm used to push into remote working dirs and
to get caught by this problem, I think I would most often escape from the
caveat, but we should take care of not trapping newbies.

> >Maybe there are other solutions too. I see that we tend to replace symlinks
> >everywhere with ref files. We might as well (in a far future version) 
> >accept
> >a file for ".git" which would contain a path to the central repo and the
> >branch's head.
> 
> There was a suggestion for something not too dissimilar even before the 
> new-workdir script:
> 
> http://thread.gmane.org/gmane.comp.version-control.git/33755

OK, thank you for your links. I still think I will wo the easy way for now,
probably using git-new-workdir, waiting for a general consensus on the subject.

Regards,
Willy

Re: Multiple working trees with GIT ?

[J. Bruce Fields]

On Thu, Jan 24, 2008 at 08:49:52AM +0100, Willy Tarreau wrote:
> Hi all,
> 
> I'm having long thoughts about how to use GIT to manage a distro. One of
> the aspects which comes very often is the notion of "variant" for a
> packaging. For instance, the whole project could consist in a list of packages
> with their branches, but this list may vary depending on the platform, the
> medium, etc... I was searching how to propagate common changes withing variants
> with the least hassle.
> 
> I figured out that having one file list per variant will be very annoying. In
> another project, that's already what I have and frankly, applying the same
> change to 10 files is counter-productive. Since the lists will often be the
> sames except for a few entries, and since most updates will be relevant to
> all variants, I thought branches will be my best friends.
> 
> But I would like to be able to always access file lists, without having to
> constantly git-checkout <variant-X>.

Could just read-only access with git-show be enough for your purposes?:

	git show <branch-name>:		# show top-level directory,
	git show <branch-name>:lib/	# show lib/ directory, etc....
	git show <branch-name>:lib/Makefile

--b.