[PATCH] Add git-bundle - pack objects and references for disconnected transfer

[PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

Some workflows require coordinated development between repositories on
machines that can never be connected. This utility creates a bundle
containing a pack of objects and associated references (heads or tags)
that can be independently transferred to another machine, effectively
supporting git-push like operations between disconnected systems.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 git-bundle.sh |  103 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 103 insertions(+), 0 deletions(-)
 create mode 100755 git-bundle.sh

diff --git a/git-bundle.sh b/git-bundle.sh
new file mode 100755
index 0000000..c431b52
--- /dev/null
+++ b/git-bundle.sh
@@ -0,0 +1,103 @@
+#!/bin/sh
+# Create a bundle to carry from one git repo to another
+# (e.g., "sneaker-net" based push)
+# git-bundle <git-rev-list args>
+# git-bundle --bare <git-rev-list args>
+# creates bundle.tar in current directory (can rename of course)
+#
+# The bundle includes all refs given (--all selects every ref in the repo).
+# and all of the commit objects needed subject to the list given.
+#
+# Objects to be packed are limited by specifying one or more of
+#   ^commit-ish    - indicated commits already at the target
+#                    (can have more than one ^commit-ish)
+#   --since=xxx    - Assume target repo has all relevant commits
+#                    earlier than xxx
+
+USAGE='git-bundle [--output=file] <git-rev-list arguments>'
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+# pull out rev-list args vs program args, parse the latter
+gitrevargs=$(git-rev-parse --symbolic --revs-only $*) || exit 1
+myargs=$(git-rev-parse --no-revs $*) || exit 1
+
+bfile=bundle.tar
+nextisoutput=
+for arg in $myargs ; do
+	case "$arg" in
+		-h|--h|--he|--hel|--help)
+			echo "$USAGE"
+			exit;;
+		--output=*)
+			bfile=${arg##--output=};;
+		*)
+			die "unknown option: $arg";;
+	esac
+done
+
+# find the refs to carry along and get sha1s for each.
+refs=
+fullrevargs=
+for arg in $gitrevargs ; do
+	#ignore options and basis refs, get unambiguous ref name for things
+	# we will transport (e.g., user gives master, have heads/master and
+	# remotes/origin/master, we keep the former).
+	case "$arg" in
+		-*) fullrevargs="$fullrevargs $arg";;
+		^*) fullrevargs="$fullrevargs $arg";;
+		*)	ref=$(git-show-ref "$arg" | head -n1)
+			[ -z "$ref" ] && die "unknown reference: $arg"
+			fullrevargs="$fullrevargs ${ref#* }"
+			refs="$refs $ref"
+			;;
+	esac
+done
+[ -z "$refs" ] && die "No references specified, I don't know what to bundle."
+
+# git-rev-list cannot determine edge objects if a date restriction is given...
+# we do things a slow way if max-age or min-age are given
+fast=
+[ "${fullrevargs##*--max-age}" == "$fullrevargs" ] && \
+[ "${fullrevargs##*--min-age}" == "$fullrevargs" ] && fast=1
+
+if [ -z "$fast" ] ; then
+	# get a list of all commits that will be packed along with parents of each.
+	# A fixed git-rev-list --boundary should replace all of this.
+	echo "Finding prerequisites and commits to bundle..."
+	commits=$(git-rev-list $fullrevargs)
+	
+	# get immediate parents of each commit to include
+	parents=
+	for c in $commits ; do
+		parents="$parents $(git-rev-list --parents $c | head -1 | cut -b42-)"
+	done
+	parents=$(printf "%s\n" $parents | sort | uniq)
+	
+	# factor out what will be in this bundle, the remainder are the bundle's pre-requisites.
+	# double up commits in this as we only want things that are only in parents to appear once
+	prereqs=$(printf "%s\n" $parents $commits $commits | \
+		sort | \
+		uniq -c | \
+		grep ' 1 ' \
+		| sed 's/ *1 //')
+else
+	prereqs=$(git-rev-list --objects-edge $fullrevargs | \
+		grep '^-' | sed 's/-//')
+fi
+
+# create refs and pack
+[ -e "$bfile" ] && rm -f "$bfile" 2>/dev/null
+printf "%s\n" $prereqs > .gitBundleReferences
+echo "-" >> .gitBundleReferences
+git-show-ref $refs >> .gitBundleReferences
+(git-rev-list --objects $fullrevargs | \
+	cut -b-40 | \
+	git pack-objects --all-progress --stdout >.gitBundlePack) \
+	|| (rm -f "$bfile" ; exit)
+tar cf "$bfile"  .gitBundleReferences .gitBundlePack
+tar cf "$bfile"  .gitBundleReferences .gitBundlePack
+rm .gitBundleReferences .gitBundlePack
+
+# done
+echo "Created $bfile"
-- 
1.5.0.34.g6afaa

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

> +# find the refs to carry along and get sha1s for each.
> +refs=
> +fullrevargs=
> +for arg in $gitrevargs ; do
> +	#ignore options and basis refs, get unambiguous ref name for things
> +	# we will transport (e.g., user gives master, have heads/master and
> +	# remotes/origin/master, we keep the former).

I am not sure if that is a good behaviour.  What if user gives
xyzzy and there are tags/xyzzy and remotes/origin/xyzzy?

I suspect it would be much safer to error out in ambiguous cases.

> +# we do things a slow way if max-age or min-age are given
> +fast=
> +[ "${fullrevargs##*--max-age}" == "$fullrevargs" ] && \
> +[ "${fullrevargs##*--min-age}" == "$fullrevargs" ] && fast=1

Style.

Our scripts tend to spell the test command "test" and equality test
operator as single '='.  Also they tend to say:

        case "$fullrevargs" in
        *--max-age* | *--min-age*) ... 
        esac

> +	# get immediate parents of each commit to include
> +	parents=
> +	for c in $commits ; do
> +		parents="$parents $(git-rev-list --parents $c | head -1 | cut -b42-)"
> +	done

Nicely done, but you seemed to have used "head -n 1" elsewhere,
which is more portable.

> +	# factor out what will be in this bundle, the remainder are the bundle's pre-requisites.
> +	# double up commits in this as we only want things that are only in parents to appear once
> +	prereqs=$(printf "%s\n" $parents $commits $commits | \
> +		sort | \
> +		uniq -c | \
> +		grep ' 1 ' \
> +		| sed 's/ *1 //')

Don't pipe output of grep to sed ;-).

> +else
> +	prereqs=$(git-rev-list --objects-edge $fullrevargs | \
> +		grep '^-' | sed 's/-//')
> +fi

Likewise.  The latter is:

	... | sed -ne 's/^-//p'

> +# create refs and pack
> +[ -e "$bfile" ] && rm -f "$bfile" 2>/dev/null
> +printf "%s\n" $prereqs > .gitBundleReferences

Our scripts assume $GIT_DIR/ is writable but not necessarily the
working directory when they have to use temporary files.  Also
please clean temporaries even when you error out with traps.

	tmp=$GIT_DIR/bundle_tmp$$
	references="$tmp-references"
        pack="$tmp-pack"
        trap 'rm -f "$tmp-*"' 0 1 2 3 15

        echo "-" >>"$references" &&
        git-show-ref $refs >>"$references" &&
        (git-rev-list ... >"$pack") || exit

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Shawn O. Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Mark Levedahl <mdl123@verizon.net> writes:
> > +# we do things a slow way if max-age or min-age are given
> > +fast=
> > +[ "${fullrevargs##*--max-age}" == "$fullrevargs" ] && \
> > +[ "${fullrevargs##*--min-age}" == "$fullrevargs" ] && fast=1
> 
> Style.
> 
> Our scripts tend to spell the test command "test" and equality test
> operator as single '='.

Using "test" is a style thing, but using "=" rather than "==" is
a portability issue.  "==" is accepted by bash as a synonym for
"=", but its not valid elsewhere.

I've made my fair share of patches with "==" in them, only to have
someone else have a problem on their system and then submit a fix to
change them to "=".  Just say no to "==" in Git shell scripts.  ;-)
 
> > +	# get immediate parents of each commit to include
> > +	parents=
> > +	for c in $commits ; do
> > +		parents="$parents $(git-rev-list --parents $c | head -1 | cut -b42-)"
> > +	done
> 
> Nicely done, but you seemed to have used "head -n 1" elsewhere,
> which is more portable.

What about:

	parents="$parents $(git-rev-list --max-count=1 --parents $c | cut -b42-)"

?

I just used that trick recently.  Oh yea it was in git-gui's
GIT-VERSION-GEN script.  Though I did not know about using cut
like that...
 
-- 
Shawn.

[PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

Some workflows require coordinated development between repositories on
machines that can never be connected. This utility creates a bundle
containing a pack of objects and associated references (heads or tags)
that can be independently transferred to another machine, effectively
supporting git-push like operations between disconnected systems.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 git-bundle.sh |  112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 112 insertions(+), 0 deletions(-)
 create mode 100755 git-bundle.sh

diff --git a/git-bundle.sh b/git-bundle.sh
new file mode 100755
index 0000000..f56f499
--- /dev/null
+++ b/git-bundle.sh
@@ -0,0 +1,112 @@
+#!/bin/sh
+# Create a bundle to carry from one git repo to another
+# (e.g., "sneaker-net" based push)
+# git-bundle <git-rev-list args>
+# git-bundle --bare <git-rev-list args>
+# creates bundle.tar in current directory (can rename of course)
+#
+# The bundle includes all refs given (--all selects every ref in the repo).
+# and all of the commit objects needed subject to the list given.
+#
+# Objects to be packed are limited by specifying one or more of
+#   ^commit-ish    - indicated commits already at the target
+#                    (can have more than one ^commit-ish)
+#   --since=xxx    - Assume target repo has all relevant commits
+#                    earlier than xxx
+
+USAGE='git-bundle [-o file | --output=file] <git-rev-list arguments>'
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+bfile=bundle.tar
+args=
+while test -n "$1" ; do
+    case $1 in
+        -h|--h|--he|--hel|--help)
+            echo "$USAGE"
+            exit;;
+        --output=*)
+            bfile=${1##--output=};;
+        -o|--output)
+            shift
+            bfile=$1;;
+        *)
+            args="$args $1"
+    esac
+    shift
+done
+
+unknown=$(git-rev-parse --no-revs $args)
+test -z "$unknown" || die "unknown option: $unknown"
+gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1
+
+# find the refs to carry along and get sha1s for each.
+refs=
+fullrevargs=
+for arg in $gitrevargs ; do
+    #ignore options and basis refs, get unambiguous ref name for things
+    # we will transport (e.g., user gives master, have heads/master and
+    # remotes/origin/master, we keep the former).
+    case "$arg" in
+        -*) fullrevargs="$fullrevargs $arg";;
+        ^*) fullrevargs="$fullrevargs $arg";;
+        *)  ref=$(git-show-ref "$arg")
+            test "$(echo $ref | wc -w)" = "2" || die "Ambigous reference: $arg
+$ref"
+            fullrevargs="$fullrevargs ${ref#* }"
+            refs="$refs $ref"
+            ;;
+    esac
+done
+test -z "$refs" && die "No references specified, I don't know what to bundle."
+
+# git-rev-list cannot determine edge objects if a date restriction is
+# given...  we do things a slow way if max-age or min-age are given
+case "$fullrevargs" in
+    *--max-age* | *--min-age*)
+    # get a list of all commits that will be packed along with
+    # parents of each.  A fixed git-rev-list --boundary should
+    # replace all of this.
+    echo "Finding prerequisites and commits to bundle..."
+    commits=$(git-rev-list $fullrevargs)
+
+    # get immediate parents of each commit to include
+    parents=
+    for c in $commits ; do
+        parents="$parents $(git-rev-list --parents --max-count=1 $c | cut -b42-)"
+    done
+    parents=$(printf "%s\n" $parents | sort | uniq)
+
+    # factor out what will be in this bundle, the remainder are the
+    # bundle's prerequisites.  double up commits in this as we only
+    # want things that are only in parents to appear once
+    prereqs=$(printf "%s\n" $parents $commits $commits | \
+        sort | uniq -c | sed -ne 's/^ *1 //p');;
+    *)
+        prereqs=$(git-rev-list --objects-edge $fullrevargs | sed -ne 's/^-//p');;
+esac
+
+# create refs and pack
+tmp=$GIT_DIR/bundle_tmp$$
+references="$tmp-references"
+pack="$tmp-pack"
+trap 'rm -f "$references" "$pack"' 0 1 2 3 15
+
+echo "v1" > "$references" &&
+echo "prerequisites" >> "$references" &&
+printf "%s\n" $prereqs >> "$references" &&
+echo "references" >> "$references" &&
+git-show-ref $refs >> "$references" &&
+(git-rev-list --objects $fullrevargs | \
+    cut -b-40 | \
+    git pack-objects --all-progress --stdout >> "$pack" ) \
+    || exit
+
+# create the tar file, clean up
+tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
+    --verbose --show-transformed-names \
+    "$references" "$pack"
+rm -f "$pack" "$references"
+
+# done
+echo "Created $bfile"
-- 
1.5.0.rc4.375.gd0938-dirty

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Shawn O. Pearce]

Mark Levedahl <mdl123@verizon.net> wrote:
> +# create the tar file, clean up
> +tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
> +    --verbose --show-transformed-names \
> +    "$references" "$pack"

I'm not sure this will work on FreeBSD.  Both 5.1 and 6.1 use tar
that does not know about --absolute-names, --transform, --verbose,
or --show-transformed-names.

-- 
Shawn.

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> Using "test" is a style thing, but using "=" rather than "==" is
> a portability issue.  "==" is accepted by bash as a synonym for
> "=", but its not valid elsewhere.

True, I should have stressed that.  Thanks.  Also in the same
sense (but to a lessor degree), "test" vs "[" is also
portability issue.

> What about:
>
> 	parents="$parents $(git-rev-list --max-count=1 --parents $c | cut -b42-)"
>
> I just used that trick recently.  Oh yea it was in git-gui's
> GIT-VERSION-GEN script.  Though I did not know about using cut
> like that...

I used to stay away from 'cut' myself, although it is not too
bad for portability these days.  I would probably have done the
above with a single sed process, though.

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Simon 'corecode' Schubert]

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

Shawn O. Pearce wrote:
> Mark Levedahl <mdl123@verizon.net> wrote:
>> +# create the tar file, clean up
>> +tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
>> +    --verbose --show-transformed-names \
>> +    "$references" "$pack"
> 
> I'm not sure this will work on FreeBSD.  Both 5.1 and 6.1 use tar
> that does not know about --absolute-names, --transform, --verbose,
> or --show-transformed-names.

for portability, pax is the official choice :)  and it can even do path name modifications with -s.  but why again are we using tar there?  this data could easiliy be put in one mixed text/binary file, starting out with

#!/bin/sh
echo "This is a git bundle.  Use git-unbundle to process me." >&1
exit
### DATA ###

or so

cheers
  simon

-- 
Serve - BSD     +++  RENT this banner advert  +++    ASCII Ribbon   /"\
Work - Mac      +++  space for low €€€ NOW!1  +++      Campaign     \ /
Party Enjoy Relax   |   http://dragonflybsd.org      Against  HTML   \
Dude 2c 2 the max   !   http://golden-apple.biz       Mail + News   / \


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

>From: Simon 'corecode' Schubert <corecode@fs.ei.tum.de>
>Date: 2007/02/16 Fri AM 05:57:20 CST
>To: "Shawn O. Pearce" <spearce@spearce.org>
>Cc: Mark Levedahl <mdl123@verizon.net>, git@vger.kernel.org
>Subject: Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

>Shawn O. Pearce wrote:
>> Mark Levedahl <mdl123@verizon.net> wrote:
>>> +# create the tar file, clean up
>>> +tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
>>> +    --verbose --show-transformed-names \
>>> +    "$references" "$pack"
>> 
>> I'm not sure this will work on FreeBSD.  Both 5.1 and 6.1 use tar
>> that does not know about --absolute-names, --transform, --verbose,
>> or --show-transformed-names.
>
>for portability, pax is the official choice :)  and it can even do path name modifications with -s.  but why again are we using tar there?  this data could easiliy be put in one mixed text/binary file, starting out with
>
>#!/bin/sh
>echo "This is a git bundle.  Use git-unbundle to process me." >&1
>exit
>### DATA ###
>
>or so
>
>cheers
>  simon

... I *tried* that, and it fails under Cygwin. Apparently cygwin's bash (or something) mangles data in the pipe (99% certain it will turn out to be a latent crlf issue)...


cat "$bfile" (
    read refs...
    git index-pack --stdin
)

worked several times, it only failed twice out of 8 bundles I tried. That's just a trifle bit too high a failure rate for my taste. ;^)

Mark

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Simon 'corecode' Schubert]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

Mark Levedahl wrote:
>> #!/bin/sh
>> echo "This is a git bundle.  Use git-unbundle to process me." >&1
>> exit
>> ### DATA ###
>>
>> or so
>>
>> cheers
>>  simon
> 
> ... I *tried* that, and it fails under Cygwin. Apparently cygwin's
> bash (or something) mangles data in the pipe (99% certain it will
> turn out to be a latent crlf issue)...

you can try something like:

sed -e '1,/^##DATA##$/d' "$bundle" | git-index-pack --stdin

and see if it works better.  no need to stream the bundle completely, it can be read several times.

cheers
  simon

-- 
Serve - BSD     +++  RENT this banner advert  +++    ASCII Ribbon   /"\
Work - Mac      +++  space for low €€€ NOW!1  +++      Campaign     \ /
Party Enjoy Relax   |   http://dragonflybsd.org      Against  HTML   \
Dude 2c 2 the max   !   http://golden-apple.biz       Mail + News   / \


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Johannes Schindelin]

Hi,

On Fri, 16 Feb 2007, Mark Levedahl wrote:

> ... I *tried* that, and it fails under Cygwin. Apparently cygwin's bash 
> (or something) mangles data in the pipe (99% certain it will turn out to 
> be a latent crlf issue)...

Have you tried

	export CYGWIN=binmode

before that? (If it works, you have to make sure that other settings as 
"ntsec" are retained in that environment variable, but not "nobinmode").

Hth,
Dscho

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

Johannes Schindelin wrote:
> Hi,
>
> On Fri, 16 Feb 2007, Mark Levedahl wrote:
>
>   
>> ... I *tried* that, and it fails under Cygwin. Apparently cygwin's bash 
>> (or something) mangles data in the pipe (99% certain it will turn out to 
>> be a latent crlf issue)...
>>     
>
> Have you tried
>
> 	export CYGWIN=binmode
>
> before that? (If it works, you have to make sure that other settings as 
> "ntsec" are retained in that environment variable, but not "nobinmode").
>
> Hth,
> Dscho
In a word, yes. I tried many things with mount and CYGWIN, none fixed 
the problem. What I don't have is a simple test case I can push upstream 
to demonstrate the failure mode. However, if this is a crlf issue I 
suspect this would not work in msys either, regardless of being able to 
find the issue in Cygwin.

Mark