Cloning into an existing, non-empty directory?

Cloning into an existing, non-empty directory?

[Richard Hartmann]

Hi all,

I know that the manpage explicitly states that it's not possible to
clones into an existing, non-empty directory, but I wanted to see if
anyone is aware of a workaround or if I can convince people that is it
of use in specific circumstances.

Some of you will be keeping their RC files in git repo(s) which means
you are basically left with two options:

1) Use soft-/hardlinks
2) Change your working tree

I went for option two which looks like:

richih@roadwarrior ~/.config/vcsh/repo.d/mr.git % ls -a
.   branches        config       FETCH_HEAD  hooks  info  objects    packed-refs
..  COMMIT_EDITMSG  description  HEAD        index  logs  ORIG_HEAD  refs
richih@roadwarrior ~/.config/vcsh/repo.d/mr.git % cat config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = /home/richih/
[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = <remote>
[branch "master"]
	remote = origin
	merge = refs/heads/master
richih@roadwarrior ~/.config/vcsh/repo.d/mr.git %

This works just fine as long as you're using

  git --work-tree=$HOME \
    --git-dir=$HOME/.config/vcsh/repo.d/mr.git \
    pull

or similar on an existing, manually mangled repo. The only real
problem left is that I can't clone initially as $HOME is obviously
non-empty, resulting in

  git --work-tree=$HOME \
    --git-dir=$HOME/.config/vcsh/repo.d/mr.git \
    clone <remote> .config/vcsh/repo.d/mr.git
fatal: working tree '/home/richih' already exists.

This means I am down to cloning, creating the directories I need,
moving .git and the actual files around, making git aware that tracked
files were not deleted, just moved, and then finally being able to use
my repo.

Obviously, this would imply some sort of conflict handling during cloning.

Anyway, I would _really_ love to have a way to avoid all this manual
work and just clone into $HOME, telling git that it's OK and I know
what I am doing.


Thanks,
Richard

PS: In case you are interested in keeping your RC files (and the rest
of your life) in git, there's a vcs-home mailing list and #vcs-home on
irc.oftc.net.

Re: Cloning into an existing, non-empty directory?

[Jonathan Nieder]

Hi,

Richard Hartmann wrote:

> The only real
> problem left is that I can't clone initially as $HOME is obviously
> non-empty, resulting in
>
>   git --work-tree=$HOME \
>     --git-dir=$HOME/.config/vcsh/repo.d/mr.git \
>     clone <remote> .config/vcsh/repo.d/mr.git
> fatal: working tree '/home/richih' already exists.
>
> This means I am down to cloning, creating the directories I need,
> moving .git and the actual files around, making git aware that tracked
> files were not deleted, just moved, and then finally being able to use
> my repo.

Have you tried something like the following?

	cd $CROWDED_DIRECTORY
	GIT_DIR=/path/to/git/dir
	GIT_WORK_TREE=$(pwd)
	export GIT_DIR GIT_WORK_TREE

	git init
	git remote add origin <remote>
	git pull

Two more caveats for the road. :)

 - git does not track detailed file permissions or secret/non-secret
   status.  Please be careful about ssh private keys and so on.

 - when updating the worktree, git will unlink existing files before
   replacing them and does not fsync the result.  So one should be
   careful when tracking files that will cause major harm if they go
   missing temporarily (though I can't think of any major examples in
   $HOME offhand).

Happy hacking,
Jonathan

Re: Cloning into an existing, non-empty directory?

[Richard Hartmann]

On Fri, Apr 22, 2011 at 00:03, Jonathan Nieder <jrnieder@gmail.com> wrote:

> Have you tried something like the following?

I didn't. Thanks a lot for your feedback.

My tentative script, which is pretty ham-fisted, takes care of most
(all?) possible errors, feedback on it is very welcome.

Two issues remain, though:

1) git merge does not seem to understand either of those parameters.
This means that I have to pull, not merge, which introduces a race
condition into my safety checks. Which leads me to

2) running `git pull` as suggested results in:

fatal: /usr/lib/git-core/git-pull cannot be used without a working tree.

which can be worked around by using --work-tree and --git-dir . Not
ideal, but workeable if I accept the race condition.




#!/bin/sh
set -x

[[ $# -lt 2 ]] && echo "not enough arguments" && exit 1
[[ $# -eq 2 ]] && GIT_WORK_TREE="$HOME/killme/fgit/target-default"
[[ $# -eq 3 ]] && GIT_WORK_TREE="$3"
[[ $# -gt 3 ]] && echo "too many arguments" && exit 2

GIT_REMOTE="$1"
GIT_DIR="$2"
export GIT_REMOTE GIT_DIR GIT_WORK_TREE
mkdir -p $GIT_WORK_TREE
cd $GIT_WORK_TREE ||
  (echo "$(basename $0): fatal: could not enter $GIT_WORK_TREE" &&
   exit 20) || exit 20
git init
git remote add origin $GIT_REMOTE
git config branch.master.remote origin
git config branch.master.merge  refs/heads/master
git fetch
for i in $(git ls-tree -r origin/master | awk '{print $4}'); do
	[[ -e $i ]] &&
	  echo "$(basename $0): error: $i exists." &&
	  CONFLICT=1;
done

[[ -n $CONFLICT ]] &&
  echo "$(basename $0): Will stop after fetching and not try to merge!\n" &&
  exit 3

git --work-tree="$GIT_WORK_TREE" --git-dir="$GIT_DIR" pull
#git --work-tree="$GIT_WORK_TREE" --git-dir="$GIT_DIR" merge


As usual, any and all feedback appreciated,
thanks,
Richard

Re: Cloning into an existing, non-empty directory?

[Jonathan Nieder]

Hi again,

Richard Hartmann wrote[1]:

> 1) git merge does not seem to understand either of those parameters.

I assume you mean --work-tree and --git-dir.  But if true, that would
be a bug.

Based on the script, it looks like you are not passing a branch name
to "git merge".  Maybe that is the problem.  In any case, it would be
nice to see what command you used, what result was expected, and what
result actually occured.

Side note: git commands respect the GIT_DIR and GIT_WORK_TREE
environment variables in addition to the corresponding command-line
parameters, which can save some typing.

[...]
> 2) running `git pull` as suggested results in:
>
> fatal: /usr/lib/git-core/git-pull cannot be used without a working tree.
>
> which can be worked around by using --work-tree and --git-dir . Not
> ideal, but workeable if I accept the race condition.

Weird.  Maybe chdir-ing into the work tree will help?  (Traditionally,
git commands tend to be run from within the work tree, so I wouldn't
be too surprised if some commands do not like being run from outside.)

[1] http://thread.gmane.org/gmane.comp.version-control.git/171920/focus=171934

Re: Cloning into an existing, non-empty directory?

[Nguyen Thai Ngoc Duy]

2011/4/23 Jonathan Nieder <jrnieder@gmail.com>:
>> 2) running `git pull` as suggested results in:
>>
>> fatal: /usr/lib/git-core/git-pull cannot be used without a working tree.
>>
>> which can be worked around by using --work-tree and --git-dir . Not
>> ideal, but workeable if I accept the race condition.
>
> Weird.  Maybe chdir-ing into the work tree will help?  (Traditionally,
> git commands tend to be run from within the work tree, so I wouldn't
> be too surprised if some commands do not like being run from outside.)

Does this patch help?

http://article.gmane.org/gmane.comp.version-control.git/150986
-- 
Duy

Re: Cloning into an existing, non-empty directory?

[Jonathan Nieder]

Nguyen Thai Ngoc Duy wrote:
>> Richard Hartmann wrote:

>>> 2) running `git pull` as suggested results in:
>>>
>>> fatal: /usr/lib/git-core/git-pull cannot be used without a working tree.
[...]
> Does this patch help?
> 
> http://article.gmane.org/gmane.comp.version-control.git/150986

That's what I was thinking of but on second thought that can't be it.
Richard's script does chdir into $GIT_WORK_TREE.

So it's still puzzling, and a reduced testcase would be nice. :)

Thanks, both, and sorry for the nonsense.

Re: Cloning into an existing, non-empty directory?

[Richard Hartmann]

2011/4/23 Jonathan Nieder <jrnieder@gmail.com>:

> Richard Hartmann wrote[1]:

Nice. Is that automagic? And if yes, may I steal it?


> I assume you mean --work-tree and --git-dir.  But if true, that would
> be a bug.

With env variables set:

% git merge origin
fatal: origin - not something we can merge
%

With parameters:

% git '--work-tree=/home/richih/killme/fgit/target-default'
'--git-dir=/home/richih/killme/fgit/source' merge origin
fatal: origin - not something we can merge
%

In a normal worktree without any mangling:
% git merge origin
[...success...]
%

I think I reduced the test-case as much as I can, especially since the
script takes care of everything for you. If you simply run in on any
random git repo, you will be able to replicate this behaviour.
Unfortunately ;)


Thanks,
Richard

Re: Cloning into an existing, non-empty directory?

[Jonathan Nieder]

Hi,

Richard Hartmann wrote:

> I think I reduced the test-case as much as I can

All right, sorry to be so dense.  I tried:

	git init test && cd test
	git remote add origin ~/src/some-repo
	git fetch origin

and then:

	$ git merge origin
	fatal: origin - not something we can merge

Is this what you mean?  "git merge" is not advertised as taking the
name of a remote repository as a parameter.  The usual usage is
instead to pass a commit, as in "git merge origin/master".

I'm not sure what a good meaning for "git merge <repository>" would
be.  "git pull <repository>" does "git fetch <repository>" and then,
if one of the fetched branches from <repository> is the configured
upstream for the current branch, merges it.

Ideas?  Probably the documentation or error message could use help
from someone less blind to see what a first-time reader would see.

Regards,
Jonathan

Re: Cloning into an existing, non-empty directory?

[Andreas Schwab]

Jonathan Nieder <jrnieder@gmail.com> writes:

> I'm not sure what a good meaning for "git merge <repository>" would
> be.

It's equivalent to refs/remotes/<repository>/HEAD if that exists.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Cloning into an existing, non-empty directory?

[Jonathan Nieder]

Andreas Schwab wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>> I'm not sure what a good meaning for "git merge <repository>" would
>> be.
>
> It's equivalent to refs/remotes/<repository>/HEAD if that exists.

Ah, I should have checked gitrevisions(7).  Thank you.

Jonathan
who wonders where his origin/HEAD symref came from and if
"git remote set-head --auto origin" was ever automatic

Re: Cloning into an existing, non-empty directory?

[Richard Hartmann]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

2011/4/25 Jonathan Nieder <jrnieder@gmail.com>:

> Is this what you mean?  "git merge" is not advertised as taking the
> name of a remote repository as a parameter.  The usual usage is
> instead to pass a commit, as in "git merge origin/master".

Ugh. I must have copied the wrong command into my script and then ran
with it, trying to fix the error I was getting in the wrong place.
Sorry for that.
I am running with environment variables and without any explicit
--work-tree --git-dir as well and everything works just fine.

In case anyone is interested, I attached the working version.


Thanks a lot for your help, I really appreciate it!
Richard

[-- Attachment #2: init_vcsh.sh --]
[-- Type: application/x-sh, Size: 828 bytes --]

Re: Cloning into an existing, non-empty directory?

[Jeff King]

On Mon, Apr 25, 2011 at 03:59:13AM -0500, Jonathan Nieder wrote:

> who wonders where his origin/HEAD symref came from and if
> "git remote set-head --auto origin" was ever automatic

It is created by "clone", but not by "git remote add". The "auto" in
remote's "set-head" means "right now, look up the HEAD on the remote and
set our $remote/HEAD based on it". Not anything about being set up or
tracked automatically.

Perhaps "git remote add" should automatically do "set-head --auto", just
as if you had cloned. I think the main reason against it is that doing
so requires hitting the network, which "remote add" (without "-f") does
not otherwise need to do.

Certainly if we are doing "-f", it would not be a big deal. In other
cases, we could perhaps fill in the value on the first "fetch", but we
may need infrastructure to know when we are on the first fetch (and it
is not simply the case that the user deleted $remote/HEAD themselves).

-Peff