git pull issues...

git pull issues...

[Morten Welinder]

Minor issues with "git pull":

1. Multiple rsync call might connect to different servers (with
round-robin DNS).  The effect
   will be interesting.  One call, if possible, would be better.

2. If any rsync fails, the process should be aborted.

As long as we do have multiple calls:

3. The "grep -v '^MOTD:'" should be changed to kill blank lines after
MOTDs also.

Morten

Re: git pull issues...

[Petr Baudis]

Dear diary, on Sat, Apr 23, 2005 at 11:25:32PM CEST, I got a letter
where Morten Welinder <mwelinder@gmail.com> told me that...
> Minor issues with "git pull":
> 
> 1. Multiple rsync call might connect to different servers (with
> round-robin DNS).  The effect
>    will be interesting.  One call, if possible, would be better.

If you can do it without overwriting HEAD, please go ahead and send me
the patch. :-)

> 2. If any rsync fails, the process should be aborted.

Fixed.

> As long as we do have multiple calls:
> 
> 3. The "grep -v '^MOTD:'" should be changed to kill blank lines after
> MOTDs also.

Actually, I currently use this as a poor man's progress indicator. ;-)
*hide*

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor

Re: git pull issues...

[Morten Welinder]

> > 1. Multiple rsync call might connect to different servers (with
> > round-robin DNS).  The effect
> >    will be interesting.  One call, if possible, would be better.
> 
> If you can do it without overwriting HEAD, please go ahead and send me
> the patch. :-)

I'll have a go at it later, but something like this ought to work:

-d .rsync-git && die "previous pull failed -- cleanup"
mkdir .rsync-git || die "cannot create .rsync-git"
ln -s ../.git/objects .rsync-git/objects || die "cannot create symlink"
rsync ... --keep-dirlinks ...
...

Morten

Re: git pull issues...

[Morten Welinder]

On 4/23/05, Petr Baudis <pasky@ucw.cz> wrote:
> > 1. Multiple rsync call might connect to different servers (with
> > round-robin DNS).  The effect
> >    will be interesting.  One call, if possible, would be better.
> 
> If you can do it without overwriting HEAD, please go ahead and send me
> the patch. :-)

This ought to work.  It basically saves to a different directory while
still ignoring
all the files we have.

Morten




Index: gitpull.sh
===================================================================
--- 7de71a831508e51e0985cea173f3f7a7012c82b7/gitpull.sh  (mode:100755
sha1:6abc7f5c00fd3e082d0a34a238a53b67c38b8a7f)
+++ uncommitted/gitpull.sh  (mode:100775)
@@ -37,28 +37,32 @@
 	[ -s ".git/heads/$name" ] && orig_head=$(cat ".git/heads/$name")
 fi
 
-
-mkdir -p .git/heads
-rsyncerr=
-rsync $RSYNC_FLAGS -Lr "$uri/heads/$rembranch" ".git/heads/$name"
2>/dev/null || rsyncerr=1
-if [ "$rsyncerr" ] && [ "$rembranch" = "master" ]; then
-	rsyncerr=
-	rsync $RSYNC_FLAGS -Lr "$uri/HEAD" ".git/heads/$name" | grep -v
'^MOTD:' || rsyncerr=1
+dotgit=dot-git
+test -d "$dotgit" && die "$dotgit already exists -- please clean up."
+mkdir -p "$dotgit" "$dotgit/objects" "$dotgit/tags" \
+	|| die "failed to create $dotgit."
+
+(cd .git && find objects -type f -print0 && find tags -type f -print0) | \
+	rsync $RSYNC_FLAGS --from0 --exclude-from=- --whole-file -Lrv
"$uri/" "$dotgit" || \
+	{ rm -rf "$dotgit"; die "rsync from $uri/ failed."; }
+
+# Move new files into place
+find "$dotgit/objects" -type f -print0 | \
+	perl -n0e 'chomp; $src=$_; s{^[^/]+/}{.git/}; rename $src, $_ or die
"failed to move $src into place: $!\n";'
+find "$dotgit/tags" -type f -print0 | \
+	perl -n0e 'chomp; $src=$_; s{^[^/]+/}{.git/}; rename $src, $_ or die
"failed to move $src into place: $!\n";'
+
+mkdir -p .git/heads || die "failed to create .git/heads"
+if [ -r "$dotgit/heads/$rembranch" ]; then
+	mv "$dotgit/heads/$rembranch" ".git/heads/$name"
+elif [ "$rembranch" = "master" -a -r "$dotgit/HEAD" ]; then
+	mv "$dotgit/HEAD" ".git/heads/$name"
+else
+	rm -rf "$dotgit"
+	die "unable to get the head pointer of branch $rembranch"
 fi
-[ "$rsyncerr" ] && die "unable to get the head pointer of branch $rembranch"
-
-[ -d .git/objects ] || mkdir -p .git/objects
-# We already saw the MOTD, thank you very much.
-rsync $RSYNC_FLAGS --ignore-existing --whole-file \
-	-v -Lr "$uri/objects/." ".git/objects/." | grep -v '^MOTD:' || die
"rsync error"
-
-# FIXME: Warn about conflicting tag names?
-# XXX: We now throw stderr to /dev/null since not all repositories
-# may have tags/ and users were confused by the harmless errors.
-[ -d .git/tags ] || mkdir -p .git/tags
-rsync $RSYNC_FLAGS --ignore-existing \
-	-v -Lr "$uri/tags/." ".git/tags/." 2>/dev/null | grep -v '^MOTD:' ||
die "rsync error"
 
+rm -rf "$dotgit"
 
 new_head=$(cat ".git/heads/$name")

Re: git pull issues...

[Dan Holmsand]

Morten Welinder wrote:
> On 4/23/05, Petr Baudis <pasky@ucw.cz> wrote:
> 
>>>1. Multiple rsync call might connect to different servers (with
>>>round-robin DNS).  The effect
>>>   will be interesting.  One call, if possible, would be better.
>>
>>If you can do it without overwriting HEAD, please go ahead and send me
>>the patch. :-)
> 
> 
> This ought to work.  It basically saves to a different directory while
> still ignoring
> all the files we have.


I was just looking at the same thing. Not saying this is better, or 
anything, bit it has, perhaps, the slight advantage that downloaded 
objects aren't thrown away, so that a pull can be safely resumed. Also, 
it doesn't download anything from an url that doesn't contain ./HEAD, 
./heads or ./objects. That might be a good thing if you accidentally try 
to pull from, say, kernel.org/pub/...

/dan


Here goes:

Index: gitpull.sh
===================================================================
--- 7de71a831508e51e0985cea173f3f7a7012c82b7/gitpull.sh  (mode:100755 
sha1:6abc7f5c00fd3e082d0a34a238a53b67c38b8a7f)
+++ uncommitted/gitpull.sh  (mode:100755)
@@ -18,7 +18,7 @@
  [ "$name" ] || name=$(cat .git/tracking 2>/dev/null)
  [ "$name" ] || die "where to pull from?"
  uri=$(grep $(echo -e "^$name\t" | sed 's/\./\\./g') .git/remotes | cut 
-f 2)
-[ "$uri" ] || die "unknown remote"
+[ "$uri" ] || die "unknown remote '$name'"

  rembranch=master
  if echo "$uri" | grep -q '#'; then
@@ -38,28 +38,39 @@
  fi


-mkdir -p .git/heads
-rsyncerr=
-rsync $RSYNC_FLAGS -Lr "$uri/heads/$rembranch" ".git/heads/$name" 
2>/dev/null || rsyncerr=1
-if [ "$rsyncerr" ] && [ "$rembranch" = "master" ]; then
-	rsyncerr=
-	rsync $RSYNC_FLAGS -Lr "$uri/HEAD" ".git/heads/$name" | grep -v 
'^MOTD:' || rsyncerr=1
+mkdir -p .git/heads .git/objects .git/tags || die
+pulldir=$(mktemp -td gitpull.XXXXXX)
+[ -d "$pulldir" ] || die "failed to create temp dir"
+ln -s $(pwd)/.git/objects $(pwd)/.git/tags $pulldir || die
+mkdir $pulldir/heads || die
+
+echo pulling $uri
+
+rout=$pulldir/rsync_errors
+rsync $RSYNC_FLAGS -LKrv --whole-file --ignore-existing \
+    --include '/HEAD' --include '/heads/' --include '/tags/' \
+    --include '/objects/' --exclude='/*' \
+    "$uri/." $pulldir 2>$rout || {
+	cat $rout; die "failed to pull from $uri"
+    }
+
+if [ -f "$pulldir/heads/$rembranch" ]; then
+    cp "$pulldir/heads/$rembranch" ".git/heads/$name" || die
+elif [ master = "$rembranch" -a -f $pulldir/HEAD ]; then
+    cp $pulldir/HEAD ".git/heads/$name" || die
+else
+    echo "unable to get the head pointer of branch $rembranch" >&2
+    echo "is $uri a valid git repository?" >&2
+    die "pull failed"
  fi
-[ "$rsyncerr" ] && die "unable to get the head pointer of branch 
$rembranch"

-[ -d .git/objects ] || mkdir -p .git/objects
-# We already saw the MOTD, thank you very much.
-rsync $RSYNC_FLAGS --ignore-existing --whole-file \
-	-v -Lr "$uri/objects/." ".git/objects/." | grep -v '^MOTD:' || die 
"rsync error"
+# TODO: Check that at least the head commit exists.

-# FIXME: Warn about conflicting tag names?
-# XXX: We now throw stderr to /dev/null since not all repositories
-# may have tags/ and users were confused by the harmless errors.
-[ -d .git/tags ] || mkdir -p .git/tags
-rsync $RSYNC_FLAGS --ignore-existing \
-	-v -Lr "$uri/tags/." ".git/tags/." 2>/dev/null | grep -v '^MOTD:' || 
die "rsync error"
+rm -rf $pulldir


+# FIXME: Warn about conflicting tag names?
+
  new_head=$(cat ".git/heads/$name")

  if [ ! "$orig_head" ]; then

Re: git pull issues...

[Morten Welinder]

The two approaches are very much alike -- there are only so many ways to
skin a cat, after all.

I actually first considered the -K approach, but then realized that only very
new rsyncs support that.  My version works with rsync 2.6.2 (and maybe
earlier).  It could be made to go further back by not using -from0 which isn't
really needed here.

But I like your check-for-HEAD detail.

Morten