[PATCH] cg-pull: summarize the number of pulled objects

[PATCH] cg-pull: summarize the number of pulled objects

[Jonas Fonseca]

Show cg-pull progress by summarizing the very verbose output of the pull
backends into a continously updated line specifying the number of
objects which have already been pulled.
		     
Signed-off-by: Jonas Fonseca <fonseca@diku.dk>
---

Straight from the bloat department, perhaps, but it is nice to not have
the terminal backlog ruined and the object count is quite nice too. :)

Interesting, it counts 4950 objects when pulling over rsync and 4454
objects when pulling locally. Didn't test HTTP pulling other than to see
if the "got <sha>" lines was matched correctly.

 cg-pull |   29 ++++++++++++++++++++++++++---
 1 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/cg-pull b/cg-pull
--- a/cg-pull
+++ b/cg-pull
@@ -29,6 +29,29 @@ if echo "$uri" | grep -q '#'; then
 	uri=$(echo $uri | cut -d '#' -f 1)
 fi
 
+pull_progress() {
+	objects=0
+	last_objects=0
+
+	while read line; do
+		case "$line" in
+		link*| symlink*| \
+		[a-f0-9][a-f0-9]/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*| \
+		"got "[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*)
+			objects=$(($objects + 1));
+			echo -ne "Pulling objects: $objects\r"
+			;;
+		*)
+			if [ "$last_objects" != "$objecst" ]; then
+				last_objects=$objects
+				echo;
+			fi
+			echo "$line"
+			;;
+		esac 
+	done;
+	[ "$last_objects" != "$objecst" ] && echo
+}
 
 fetch_rsync () {
 	redir=
@@ -62,7 +85,7 @@ fetch_rsync () {
 }
 
 pull_rsync () {
-	fetch_rsync -s -u -d "$2/objects" "$_git_objects"
+	fetch_rsync -s -u -d "$2/objects" "$_git_objects" | pull_progress
 }
 
 
@@ -107,7 +130,7 @@ fetch_http () {
 }
 
 pull_http () {
-	git-http-pull -a -v "$(cat "$_git/refs/heads/$1")" "$2/"
+	(git-http-pull -a -v "$(cat "$_git/refs/heads/$1")" "$2/" 2>&1 /dev/null) | pull_progress
 }
 
 
@@ -170,7 +193,7 @@ fetch_local () {
 }
 
 pull_local () {
-	git-local-pull -a -l -v "$(cat "$_git/refs/heads/$1")" "$2"
+	(git-local-pull -a -l -v "$(cat "$_git/refs/heads/$1")" "$2" 2>&1 /dev/null) | pull_progress
 }
 
 if echo "$uri" | grep -q "^http://"; then
-- 
Jonas Fonseca

Re: [PATCH] cg-pull: summarize the number of pulled objects

[Frank Sorenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jonas Fonseca wrote:
> +			if [ "$last_objects" != "$objecst" ]; then
                                                 ^^^^^^^^
Did you mean 'objects' ???

> +				last_objects=$objects
> +				echo;
> +			fi
> +			echo "$line"
> +			;;
> +		esac 
> +	done;
> +	[ "$last_objects" != "$objecst" ] && echo
                               ^^^^^^^
Here too?

Frank
- --
Frank Sorenson - KD7TZK
Systems Manager, Computer Science Department
Brigham Young University
frank@tuxrocks.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCmo0haI0dwg4A47wRAkJHAJ98i7xUZZVd3rzXdlor9f5+Lly7SgCfc4qK
dXkHJegPZLxP3CKzvm7SFHM=
=sKlv
-----END PGP SIGNATURE-----

Re: [PATCH] cg-pull: summarize the number of pulled objects

[Jonas Fonseca]

Frank Sorenson <frank@tuxrocks.com> wrote Sun, May 29, 2005:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Jonas Fonseca wrote:
> > +			if [ "$last_objects" != "$objecst" ]; then
>                                                  ^^^^^^^^
> Did you mean 'objects' ???
> 
> > +				last_objects=$objects
> > +				echo;
> > +			fi
> > +			echo "$line"
> > +			;;
> > +		esac 
> > +	done;
> > +	[ "$last_objects" != "$objecst" ] && echo
>                                ^^^^^^^
> Here too?

Yes. Thanks.

-- 
Jonas Fonseca

Re: [PATCH] cg-pull: summarize the number of pulled objects

[Petr Baudis]

Dear diary, on Mon, May 30, 2005 at 03:56:50AM CEST, I got a letter
where Jonas Fonseca <fonseca@diku.dk> told me that...
> Show cg-pull progress by summarizing the very verbose output of the pull
> backends into a continously updated line specifying the number of
> objects which have already been pulled.
> 		     
> Signed-off-by: Jonas Fonseca <fonseca@diku.dk>
> ---
> 
> Straight from the bloat department, perhaps, but it is nice to not have
> the terminal backlog ruined and the object count is quite nice too. :)
> 
> Interesting, it counts 4950 objects when pulling over rsync and 4454
> objects when pulling locally. Didn't test HTTP pulling other than to see
> if the "got <sha>" lines was matched correctly.

It is nice, but actually losing information for me. It's ok for the HTTP
and SSH "smart pulls", but if I pull over rsync, I have progress
indication based on the first two digits - assuming roughly even
distribution of hashes, I can see that when I cross 80/ I'm in the half
etc. So what about showing some percentage or something in that case?

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor

Re: [PATCH] cg-pull: summarize the number of pulled objects

[Jonas Fonseca]

Petr Baudis <pasky@ucw.cz> wrote Mon, May 30, 2005:
> Dear diary, on Mon, May 30, 2005 at 03:56:50AM CEST, I got a letter
> where Jonas Fonseca <fonseca@diku.dk> told me that...
> > Show cg-pull progress by summarizing the very verbose output of the pull
> > backends into a continously updated line specifying the number of
> > objects which have already been pulled.
> > 		     
> > Signed-off-by: Jonas Fonseca <fonseca@diku.dk>
> > ---
> > 
> > Straight from the bloat department, perhaps, but it is nice to not have
> > the terminal backlog ruined and the object count is quite nice too. :)
> > 
> > Interesting, it counts 4950 objects when pulling over rsync and 4454
> > objects when pulling locally. Didn't test HTTP pulling other than to see
> > if the "got <sha>" lines was matched correctly.
> 
> It is nice, but actually losing information for me. It's ok for the HTTP
> and SSH "smart pulls", but if I pull over rsync, I have progress
> indication based on the first two digits - assuming roughly even
> distribution of hashes, I can see that when I cross 80/ I'm in the half
> etc. So what about showing some percentage or something in that case?

[ Ok, I've done the percentage thing. The number of bytes are also
  tracked. I also tried to return the correct exit code from the pull
  function using PIPESTATUS. ]

[PATCH] cg-pull: summarize the number of pulled objects

Show object pulling progress by summarizing the verbose output of the
pull backends into a single continously updated status line. It displays
the number of already pulled objects, the total size of the pulled
objects and for rsync the percentage done based on what object
sub-directory objects are coming from. The last thing means it might not
always get all the way up to 100%.

Signed-off-by: Jonas Fonseca <fonseca@diku.dk>

---

 cg-pull |   55 +++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 51 insertions(+), 4 deletions(-)

diff --git a/cg-pull b/cg-pull
--- a/cg-pull
+++ b/cg-pull
@@ -29,6 +29,49 @@ if echo "$uri" | grep -q '#'; then
 	uri=$(echo $uri | cut -d '#' -f 1)
 fi
 
+pull_progress() {
+	percentage=""
+	objects=0
+	last_objects=0
+	size=0
+
+	while read line; do
+		object=
+
+		case "$line" in
+		link*|symlink*|\
+		"got "[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*)
+			# Convert the sha to an object path
+			object=$(echo "$line" | sed 's,.* \([a-f0-9][a-f0-9]\),\1/,')
+			;;
+
+		[a-f0-9][a-f0-9]/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*)
+			object="$line"
+			# Estimate percentage done using the position of
+			# the object subdir. It might not get all the way
+			# up to 100% ...
+			position=$(echo "$line" | cut -d/ -f 1)
+			percentage=", $((0x$position * 100 / 0xff))% done"
+			;;
+
+		*)
+			if [ "$last_objects" != "$objects" ]; then
+				last_objects="$objects"
+				echo;
+			fi
+			echo "$line"
+			continue
+			;;
+		esac 
+
+		object="$_git_objects/$object"
+		size=$(($size + $(stat -c '%s' "$object" 2>/dev/null)))
+		objects=$(($objects + 1));
+
+		echo -ne "progress: $objects objects, $size bytes$percentage\r"
+	done;
+	[ "$last_objects" != "$objects" ] && echo
+}
 
 fetch_rsync () {
 	redir=
@@ -62,7 +105,8 @@ fetch_rsync () {
 }
 
 pull_rsync () {
-	fetch_rsync -s -u -d "$2/objects" "$_git_objects"
+	fetch_rsync -s -u -d "$2/objects" "$_git_objects" | pull_progress
+	return ${PIPESTATUS[0]}
 }
 
 
@@ -107,7 +151,8 @@ fetch_http () {
 }
 
 pull_http () {
-	git-http-pull -a -v "$(cat "$_git/refs/heads/$1")" "$2/"
+	(git-http-pull -a -v "$(cat "$_git/refs/heads/$1")" "$2/" 2>&1 /dev/null) | pull_progress
+	return ${PIPESTATUS[0]}
 }
 
 
@@ -142,7 +187,8 @@ fetch_ssh () {
 }
 
 pull_ssh () {
-	git-rpull -a -v "$(cat "$_git/refs/heads/$1")" "$2"
+	(git-rpull -a -v "$(cat "$_git/refs/heads/$1")" "$2" 2>&1 /dev/null) | pull_progress
+	return ${PIPESTATUS[0]}
 }
 
 
@@ -170,7 +216,8 @@ fetch_local () {
 }
 
 pull_local () {
-	git-local-pull -a -l -v "$(cat "$_git/refs/heads/$1")" "$2"
+	(git-local-pull -a -l -v "$(cat "$_git/refs/heads/$1")" "$2" 2>&1 /dev/null) | pull_progress
+	return ${PIPESTATUS[0]}
 }
 
 if echo "$uri" | grep -q "^http://"; then
-- 
Jonas Fonseca