Why is git-clone --reference so slow?

Why is git-clone --reference so slow?

[David Woodhouse]

My DSL line sucks; I know this. But why is git-clone so bad at using it?

This is slow and seems to be downloading a lot of stuff it doesn't
need...

pmac /pmac/git $ git-clone --reference /pmac/git/linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git
remote: Generating pack...
remote: Done counting 14428 objects.
remote: Deltifying 14428 objects.
remote:  100% (14428/14428) done
Indexing 14428 objects.
  25% (3662/14428) done

At this point I got bored and hit ^C, then logged in to hera and did it
there, then rsync'd the result...

hera /home/dwmw2/x $ git-clone -n --bare --reference /pub/scm/linux/kernel/git/torvalds/linux-2.6.git /pub/scm/linux/kernel/git/geoff/ps3-linux.git 
remote: Generating pack...
remote: Done counting 14490 objects.
remote: Deltifying 14490 objects.
remote:  100% (14490/14490) done
Indexing 14490 objects.
remote: Total 14490, written 14490 (delta 579), reused 5598 (delta 367)
 100% (14490/14490) done
Resolving 579 deltas.
 100% (579/579) done

pmac /pmac/git/ps3-linux $  rsync -avz master.kernel.org:/home/dwmw2/x/ps3-linux/ .git
pmac /pmac/git/ps3-linux $ echo /pmac/git/linux-2.6/.git/objects > .git/objects/info/alternates

That was a _lot_ faster, and abused my poor DSL line a lot less. Why's
git so bad at it?

-- 
dwmw2

Re: Why is git-clone --reference so slow?

[Johannes Schindelin]

Hi,

On Mon, 5 Feb 2007, David Woodhouse wrote:

> My DSL line sucks; I know this. But why is git-clone so bad at using it?
> 
> This is slow and seems to be downloading a lot of stuff it doesn't
> need...
> 
> pmac /pmac/git $ git-clone --reference /pmac/git/linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git

Is this a Mac? Is it Mac OS X? Which git version do you use?

We recently had a case like this, and it was fixed: apparently, mmap() on 
Mac OS X is less than fast...

Hth,
Dscho

Re: Why is git-clone --reference so slow?

[David Woodhouse]

On Mon, 2007-02-05 at 16:31 +0100, Johannes Schindelin wrote:
> Is this a Mac? Is it Mac OS X? Which git version do you use?
> 
> We recently had a case like this, and it was fixed: apparently, mmap()
> on Mac OS X is less than fast... 

Yes, it's a Mac G5. No, it's not OSX; it's Fedora Core 6. As far as I
can tell it's downloading far more than it needs before discarding
objects which are already in the local repository used in --reference.


-- 
dwmw2

Re: Why is git-clone --reference so slow?

[Junio C Hamano]

David Woodhouse <dwmw2@infradead.org> writes:

> My DSL line sucks; I know this. But why is git-clone so bad at using it?

I did the same and it initially got 45MB pack (14.5k objects).

For my case, the reason was because the repository I used as the
reference was pruned and packed its refs, and git-clone was not
fully using the refs from the reference repository.

The following patch made it to be almost instantaneous.  The
resulting clone has 600kB pack (0.5k objects).


diff --git a/git-clone.sh b/git-clone.sh
index 1710996..1bd54de 100755
--- a/git-clone.sh
+++ b/git-clone.sh
@@ -178,46 +178,32 @@ esac && export GIT_DIR && git-init ${template+"$template"} || usage
 
 if test -n "$reference"
 then
+	ref_git=
 	if test -d "$reference"
 	then
 		if test -d "$reference/.git/objects"
 		then
-			reference="$reference/.git"
+			ref_git="$reference/.git"
+		elif test -d "$reference/objects"
+		then
+			ref_git="$reference"
 		fi
-		reference=$(cd "$reference" && pwd)
-		echo "$reference/objects" >"$GIT_DIR/objects/info/alternates"
-		(cd "$reference" && tar cf - refs) |
-		(cd "$GIT_DIR/refs" &&
-		 mkdir reference-tmp &&
-		 cd reference-tmp &&
-		 tar xf - &&
-		 find refs ! -type d -print |
-		 while read ref
-		 do
-			if test -h "$ref"
-			then
-				# Old-style symbolic link ref.  Not likely
-				# to appear under refs/ but we might as well
-				# deal with them.
-				:
-			elif test -f "$ref"
-			then
-				point=$(cat "$ref") &&
-					case "$point" in
-					'ref: '*) ;;
-					*) continue ;;
-					esac
-			fi
-			# The above makes true ref to 'continue' and
-			# we will come here when we are looking at
-			# symbolic link ref or a textual symref (or
-			# garbage, like fifo).
-			# The true ref pointed at by it is enough to
-			# ensure that we do not fetch objects reachable
-			# from it.
-			rm -f "$ref"
-		 done
-		)
+	fi
+	if test -n "$ref_git"
+	then
+		ref_git=$(cd "$ref_git" && pwd)
+		echo "$ref_git/objects" >"$GIT_DIR/objects/info/alternates"
+		(
+			GIT_DIR="$ref_git" git for-each-ref \
+				--format='%(objectname) %(*objectname)'
+		) |
+		while read a b
+		do
+			test -z "$a" ||
+			git update-ref "refs/reference-tmp/$a" "$a"
+			test -z "$b" ||
+			git update-ref "refs/reference-tmp/$b" "$b"
+		done
 	else
 		die "reference repository '$reference' is not a local directory."
 	fi