expensive local git clone

expensive local git clone

[David S. Miller]

I'm still seeing git-local-pull going crazy when attemping local
clones of kernel trees, this is from top:

22032 davem     18   0 63392  59m  920 R 76.0  5.9   5:33.15 git-local-pull     

and it's still chugging along.  This is from a simple:

bash$ git-clone-script /home/davem/src/GIT/linux-2.6/.git /home/davem/src/GIT/sparc-2.6

invocation.

I can't be the only person in the world seeing this :-)
It usually grows to around 88MB and runs for 8 minutes before
it finishes.

It must be walking the whole commit tree to the root or something
silly like that.

Re: expensive local git clone

[Linus Torvalds]

On Mon, 4 Jul 2005, David S. Miller wrote:
> 
> I can't be the only person in the world seeing this :-)
> It usually grows to around 88MB and runs for 8 minutes before
> it finishes.

I'm making something based on pack-files.

git-local-pull is indeed walking the whole tree and copying one file at a
time. We shouldn't use "git-fetch-script" in "git clone" (it's really
designed for doing one file at a time and for small updates),

That said, the pack-file thing I'm working on won't be perfect either, and
the main advantage is that it should work over ssh. The fastest way to do
a clone is really to do a recursive hardlinked tree, so you may well want
to just do

	#!/bin/sh
	mkdir "$2" && cp -rl "$1/.git" "$2/.git"

and it should be about a million times faster and equally effective.

Untested, of course.

		Linus

Re: expensive local git clone

[Petr Baudis]

Dear diary, on Mon, Jul 04, 2005 at 10:39:09PM CEST, I got a letter
where Linus Torvalds <torvalds@osdl.org> told me that...
> That said, the pack-file thing I'm working on won't be perfect either, and
> the main advantage is that it should work over ssh. The fastest way to do
> a clone is really to do a recursive hardlinked tree, so you may well want
> to just do
> 
> 	#!/bin/sh
> 	mkdir "$2" && cp -rl "$1/.git" "$2/.git"
> 
> and it should be about a million times faster and equally effective.
> 
> Untested, of course.

I wouldn't do that on anything but .git/objects. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..

Re: expensive local git clone

[David S. Miller]

From: Petr Baudis <pasky@suse.cz>
Date: Mon, 4 Jul 2005 22:42:35 +0200

> Dear diary, on Mon, Jul 04, 2005 at 10:39:09PM CEST, I got a letter
> where Linus Torvalds <torvalds@osdl.org> told me that...
> > That said, the pack-file thing I'm working on won't be perfect either, and
> > the main advantage is that it should work over ssh. The fastest way to do
> > a clone is really to do a recursive hardlinked tree, so you may well want
> > to just do
> > 
> > 	#!/bin/sh
> > 	mkdir "$2" && cp -rl "$1/.git" "$2/.git"
> > 
> > and it should be about a million times faster and equally effective.
> > 
> > Untested, of course.
> 
> I wouldn't do that on anything but .git/objects. ;-)

Agreed.  When I do it by hand (but I shouldn't have to
do this by hand, git-clone-script should do it this way
if that's the best and most efficient) I usually link the
object directory then copy the rest of the stuff non-linked.

I keep hoping git-clone-script is going to be a good way
to clone two local trees.  Is my hope misguided?  :-)

Re: expensive local git clone

[Junio C Hamano]

>>>>> "DSM" == David S Miller <davem@davemloft.net> writes:

DSM> I keep hoping git-clone-script is going to be a good way
DSM> to clone two local trees.  Is my hope misguided?  :-)

Something along these lines?
------------
Short-cut "git fetch" when cloning locally.

Instead of calling local-pull to "do the right thing", just
hardlink or copy the object files over when we know we are doing
the clone locally.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

cd /opt/packrat/playpen/public/in-place/git/git.junio/
jit-diff
# - master: git-format-patch: Prepare patches for e-mail submission.
# + (working tree)
diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,7 +1,39 @@
 #!/bin/sh
 repo="$1"
 dir="$2"
-mkdir $dir || exit 1
-cd $dir
-git-init-db
-git fetch "$repo" && ( git-rev-parse FETCH_HEAD > .git/HEAD )
+mkdir "$dir" &&
+D=$(
+    (cd "$dir" && git-init-db && pwd)
+) || exit 1
+
+# See if repo is a local directory.
+if (
+	cd "$repo/objects" 2>/dev/null
+)
+then
+	# See if we can hardlink and drop "l" if not.
+	sample_file=$(cd "$repo" && find objects -type f -print | sed -e 1q)
+
+	# objects directory should not be empty since we are cloning!
+	test -f "$repo/$sample_file" || exit 
+
+	if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	then
+	    l=l
+	else
+	    l=
+	fi &&
+	rm -f "$D/.git/objects/sample" &&
+
+	cd "$repo" &&
+	find objects -type f -print |
+	cpio -puam$l "$D/.git" || exit 1
+
+	# FETCH_HEAD is always HEAD because we do not do the
+	# extra parameter to "git fetch".
+	pwd
+	cat "HEAD" >"$D/.git/FETCH_HEAD"
+	cd "$D"
+else
+	cd "$D" && git fetch "$repo"
+fi && ( git-rev-parse FETCH_HEAD > .git/HEAD )

Compilation finished at Mon Jul  4 14:37:29

Re: expensive local git clone

[David S. Miller]

From: Junio C Hamano <junkio@cox.net>
Date: Mon, 04 Jul 2005 14:40:11 -0700

> >>>>> "DSM" == David S Miller <davem@davemloft.net> writes:
> 
> DSM> I keep hoping git-clone-script is going to be a good way
> DSM> to clone two local trees.  Is my hope misguided?  :-)
> 
> Something along these lines?

Looks interesting.  Any particular reason to use "cpio"
instead of "cp"?

Re: expensive local git clone

[Junio C Hamano]

>>>>> "DSM" == David S Miller <davem@davemloft.net> writes:

DSM> Looks interesting.  Any particular reason to use "cpio"
DSM> instead of "cp"?

No particular reason other than I am just used to doing things
that way.

Re: expensive local git clone

[Linus Torvalds]

On Mon, 4 Jul 2005, David S. Miller wrote:
> 
> I keep hoping git-clone-script is going to be a good way
> to clone two local trees.  Is my hope misguided?  :-)

Well, I'm not working on it, but tested patches...

		Linus

[PATCH] Short-cut git-fetch-pack while cloning locally.

[Junio C Hamano]

>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:

LT> Well, I'm not working on it, but tested patches...

Prodded by the hint...

------------
The git-fetch-pack command used internally by git-fetch-script
which in turn is used by git-clone-script refuses to run if
there is no common head.  When cloning locally, just copy/link the
objects directory to avoid its use, since we know there will not
be any common head (the target is empty).  It also should be
more efficient.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

*** This is slightly different from the one I sent as a response
*** to David's inquiry, in that it uses cp -rl (or just cp -r)
*** instead of cpio.  Also I added the copyright notices and one
*** liner description.

 git-clone-script |   43 +++++++++++++++++++++++++++++++++++++++----
 rev-list.c       |    5 ++++-
 2 files changed, 43 insertions(+), 5 deletions(-)

f33df082e175f7fc717a75e528eab3db1d294576
diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,7 +1,42 @@
 #!/bin/sh
+#
+# Copyright (c) 2005, Linus Torvalds
+# Copyright (c) 2005, Junio C Hamano
+# 
+# Clone a repository into a different directory that does not yet exist.
+
 repo="$1"
 dir="$2"
-mkdir $dir || exit 1
-cd $dir
-git-init-db
-git fetch "$repo" && ( git-rev-parse FETCH_HEAD > .git/HEAD )
+mkdir "$dir" &&
+D=$(
+	(cd "$dir" && git-init-db && pwd)
+) || exit 1
+
+# See if repo is a local directory.
+if (
+	cd "$repo/objects" 2>/dev/null
+)
+then
+	# See if we can hardlink and drop "l" if not.
+	sample_file=$(cd "$repo" && find objects -type f -print | sed -e 1q)
+
+	# objects directory should not be empty since we are cloning!
+	test -f "$repo/$sample_file" || exit
+
+	if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	then
+		l=l
+	else
+		l=
+	fi &&
+	rm -f "$D/.git/objects/sample" &&
+
+	cp -r$l "$repo/objects" "$D/.git/" || exit 1
+
+	# FETCH_HEAD is always HEAD because we do not do the
+	# extra parameter to "git fetch".
+	cat "$repo/HEAD" >"$D/.git/FETCH_HEAD"
+	cd "$D"
+else
+	cd "$D" && git fetch "$repo"
+fi && git-rev-parse FETCH_HEAD > .git/HEAD
diff --git a/rev-list.c b/rev-list.c
--- a/rev-list.c
+++ b/rev-list.c
@@ -482,7 +482,10 @@ int main(int argc, char **argv)
 		commit = get_commit_reference(arg, flags);
 		if (!commit)
 			continue;
-		insert_by_date(&list, commit);
+		if (!merge_order)
+			insert_by_date(&list, commit);
+		else
+			commit_list_insert(commit, &list);
 	}
 
 	if (!merge_order) {		

Re: expensive local git clone

[Linus Torvalds]

On Mon, 4 Jul 2005, Linus Torvalds wrote:
> 
> I'm making something based on pack-files.

Ok, as of a few minutes ago there's now a pack-based "git clone" there. Of 
course, it hasn't percolated out to the mirrors yet, so you'll not see it 
for a while unless you have a master.kernel.org account like David, but 
basically you can do

	git clone /some/absolute/path new-dir

and it will clone the old project into new-dir using pack-files. The need 
for an absolute path in the source is just because the "git clone" script 
ends up "cd'ing" to the new directory before the clone, so a relative path 
would need to be relative to the "new-dir" location, which is just 
confusing. Oh, well.

Anyway, as mentioned, you can certainly do a local clone a lot faster with 
"cp -rl" (and yes, I'll apply Junio's patch if he makes it available 
against the new version, and adds a flag to make it conditional), but 
using a pack-file means that the new thing not only will be totally 
independent of the old one (which you may _want_ to do, especially if they 
are on different filesystems), but it also means that the above works over 
ssh too, ie

	git clone master.kernel.org:/pub/scm/git/git.git my-git

should now do the right thing (indeed, I've even tested it - although I've
not tried it with things like multiple branches etc, which _should_ all
work automatically but.. I'm pulling Jeff's tree as I write this, but 
master.kernel.org is slow, so it will be some time..).

Btw, one small note: when doing a "git clone", the newly cloned repo will 
not be checked out, and "master" will be the default HEAD regardless of 
what the other happened to be at (unless the other side was really screwed 
up and we can't match up any "master" at all). So use "git checkout xxx" 
to actually set whatever branch you want to use after a clone.

		Linus

[PATCH] Short-circuit git-clone-pack while cloning locally.

[Junio C Hamano]

>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:

LT> Anyway, as mentioned, you can certainly do a local clone a lot faster with 
LT> "cp -rl" (and yes, I'll apply Junio's patch if he makes it available 
LT> against the new version, and adds a flag to make it conditional),...

By invitation.

------------
When we are cloning a repository on a local filesystem, it is
faster to just create a hard linkfarm of .git/object hierarchy
and copy the .git/refs files.  By default, the script uses the
clone-pack method, but with -l and -c parameters it can be told
to do the hard linkfarm and recursive file copy to replicate
.git/object hierarchy.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 git-clone-script |   80 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 79 insertions(+), 1 deletions(-)

1eb81b1791a751ca0fd1c3e7c1d4a4baacdd042a
diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,4 +1,81 @@
 #!/bin/sh
+#
+# Copyright (c) 2005, Linus Torvalds
+# Copyright (c) 2005, Junio C Hamano
+# 
+# Clone a repository into a different directory that does not yet exist.
+
+usage() {
+	echo >&2 "* git clone [-l|-c|-p] <repo> <dir>"
+	exit 1
+}
+
+# what to do when running locally
+local_use=default
+while
+	case "$#,$1" in
+	0,*) break ;;
+	*,-l) local_use=link ;;
+	*,-c) local_use=copy ;;
+	*,-p) local_use=pack ;;
+	*,-*) usage ;;
+	*) break ;;
+	esac
+do
+	shift
+done
+
 repo="$1"
 dir="$2"
-mkdir "$dir" && cd "$dir" && git-init-db && git-clone-pack "$repo"
+mkdir "$dir" &&
+D=$(
+	(cd "$dir" && git-init-db && pwd)
+) || exit 1
+
+# See if repo is a local directory.
+is_local=f
+if (
+	cd "$repo/objects" 2>/dev/null
+)
+then
+	is_local=t
+fi
+
+case "$local_use,$is_local" in
+default,f)
+	;;
+*,f)	echo >&2 "-l|-c|-p flag can be used only when cloning a local repo"
+	exit 1
+	;;
+copy,t | link,t)
+	l=
+	case "$local_use" in
+	link)
+	    # See if we can hardlink and drop "l" if not.
+	    sample_file=$(cd "$repo" && \
+	    		  find objects -type f -print | sed -e 1q)
+
+	    # objects directory should not be empty since we are cloning!
+	    test -f "$repo/$sample_file" || exit
+
+	    if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	    then
+		    l=l
+	    fi &&
+	    rm -f "$D/.git/objects/sample"
+	    ;;
+	esac &&
+	cp -r$l "$repo/objects" "$D/.git/" || exit 1
+
+	# Make a duplicate of refs and HEAD pointer
+	HEAD=
+	if test -f "$repo/HEAD"
+	then
+		HEAD=HEAD
+	fi
+	tar Ccf "$repo" - refs $HEAD | tar Cxf "$D/.git" - || exit 1
+	;;
+*)
+	cd "$D" && git clone-pack "$repo"
+	;;
+esac

Re: [PATCH] Short-circuit git-clone-pack while cloning locally.

[Linus Torvalds]

On Wed, 6 Jul 2005, Junio C Hamano wrote:
> 
> By invitation.
> 
> ------------
> When we are cloning a repository on a local filesystem [...]

Hmm.. Did you test the ssh case?

> +case "$local_use,$is_local" in
> +default,f)
> +	;;

It would seem that you don't do anything at all for the non-local case.

		Linus

Re: [PATCH] Short-circuit git-clone-pack while cloning locally.

[Junio C Hamano]

>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:

LT> It would seem that you don't do anything at all for the non-local case.

Oops.

[PATCH] Short-circuit git-clone-pack while cloning locally (take 2).

[Junio C Hamano]

When we are cloning a repository on a local filesystem, it is
faster to just create a hard linkfarm of .git/object hierarchy
and copy the .git/refs files.  By default, the script uses the
clone-pack method, but it can be told with the -l flag to do the
hard linkfarm (falling back on recursive file copy) to replicate
the .git/object hierarchy.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 git-clone-script |   68 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 67 insertions(+), 1 deletions(-)

2bff57b39b9fd13b1319d0192e2a887eb82fa189
diff --git a/git-clone-script b/git-clone-script
--- a/git-clone-script
+++ b/git-clone-script
@@ -1,4 +1,70 @@
 #!/bin/sh
+#
+# Copyright (c) 2005, Linus Torvalds
+# Copyright (c) 2005, Junio C Hamano
+# 
+# Clone a repository into a different directory that does not yet exist.
+
+usage() {
+	echo >&2 "* git clone [-l] <repo> <dir>"
+	exit 1
+}
+
+use_local=no
+while
+	case "$#,$1" in
+	0,*) break ;;
+        *,-l|*,--l|*,--lo|*,--loc|*,--loca|*,--local) use_local=yes ;;
+	*,-*) usage ;;
+	*) break ;;
+	esac
+do
+	shift
+done
+
 repo="$1"
 dir="$2"
-mkdir "$dir" && cd "$dir" && git-init-db && git-clone-pack "$repo"
+mkdir "$dir" &&
+D=$(
+	(cd "$dir" && git-init-db && pwd)
+) &&
+test -d "$D" || usage
+
+# We do local magic only when the user tells us to.
+case "$use_local" in
+yes)
+	( cd "$repo/objects" ) || {
+		repo="$repo/.git"
+		( cd "$repo/objects" ) || {
+		    echo >&2 "-l flag seen but $repo is not local."
+		    exit 1
+		}
+	}
+
+	# See if we can hardlink and drop "l" if not.
+	sample_file=$(cd "$repo" && \
+		      find objects -type f -print | sed -e 1q)
+
+	# objects directory should not be empty since we are cloning!
+	test -f "$repo/$sample_file" || exit
+
+	l=
+	if ln "$repo/$sample_file" "$D/.git/objects/sample" 2>/dev/null
+	then
+		l=l
+	fi &&
+	rm -f "$D/.git/objects/sample" &&
+	cp -r$l "$repo/objects" "$D/.git/" || exit 1
+
+	# Make a duplicate of refs and HEAD pointer
+	HEAD=
+	if test -f "$repo/HEAD"
+	then
+		HEAD=HEAD
+	fi
+	tar Ccf "$repo" - refs $HEAD | tar Cxf "$D/.git" - || exit 1
+	exit 0
+	;;
+esac
+
+cd "$D" && git clone-pack "$repo"