Re: Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

Re: Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

>From: Simon 'corecode' Schubert <corecode@fs.ei.tum.de>
>Date: 2007/02/16 Fri AM 05:57:20 CST
>To: "Shawn O. Pearce" <spearce@spearce.org>
>Cc: Mark Levedahl <mdl123@verizon.net>, git@vger.kernel.org
>Subject: Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

>Shawn O. Pearce wrote:
>> Mark Levedahl <mdl123@verizon.net> wrote:
>>> +# create the tar file, clean up
>>> +tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
>>> +    --verbose --show-transformed-names \
>>> +    "$references" "$pack"
>> 
>> I'm not sure this will work on FreeBSD.  Both 5.1 and 6.1 use tar
>> that does not know about --absolute-names, --transform, --verbose,
>> or --show-transformed-names.
>
>for portability, pax is the official choice :)  and it can even do path name modifications with -s.  but why again are we using tar there?  this data could easiliy be put in one mixed text/binary file, starting out with
>
>#!/bin/sh
>echo "This is a git bundle.  Use git-unbundle to process me." >&1
>exit
>### DATA ###
>
>or so
>
>cheers
>  simon

... I *tried* that, and it fails under Cygwin. Apparently cygwin's bash (or something) mangles data in the pipe (99% certain it will turn out to be a latent crlf issue)...


cat "$bfile" (
    read refs...
    git index-pack --stdin
)

worked several times, it only failed twice out of 8 bundles I tried. That's just a trifle bit too high a failure rate for my taste. ;^)

Mark

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Simon 'corecode' Schubert]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

Mark Levedahl wrote:
>> #!/bin/sh
>> echo "This is a git bundle.  Use git-unbundle to process me." >&1
>> exit
>> ### DATA ###
>>
>> or so
>>
>> cheers
>>  simon
> 
> ... I *tried* that, and it fails under Cygwin. Apparently cygwin's
> bash (or something) mangles data in the pipe (99% certain it will
> turn out to be a latent crlf issue)...

you can try something like:

sed -e '1,/^##DATA##$/d' "$bundle" | git-index-pack --stdin

and see if it works better.  no need to stream the bundle completely, it can be read several times.

cheers
  simon

-- 
Serve - BSD     +++  RENT this banner advert  +++    ASCII Ribbon   /"\
Work - Mac      +++  space for low €€€ NOW!1  +++      Campaign     \ /
Party Enjoy Relax   |   http://dragonflybsd.org      Against  HTML   \
Dude 2c 2 the max   !   http://golden-apple.biz       Mail + News   / \


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Johannes Schindelin]

Hi,

On Fri, 16 Feb 2007, Mark Levedahl wrote:

> ... I *tried* that, and it fails under Cygwin. Apparently cygwin's bash 
> (or something) mangles data in the pipe (99% certain it will turn out to 
> be a latent crlf issue)...

Have you tried

	export CYGWIN=binmode

before that? (If it works, you have to make sure that other settings as 
"ntsec" are retained in that environment variable, but not "nobinmode").

Hth,
Dscho

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

Johannes Schindelin wrote:
> Hi,
>
> On Fri, 16 Feb 2007, Mark Levedahl wrote:
>
>   
>> ... I *tried* that, and it fails under Cygwin. Apparently cygwin's bash 
>> (or something) mangles data in the pipe (99% certain it will turn out to 
>> be a latent crlf issue)...
>>     
>
> Have you tried
>
> 	export CYGWIN=binmode
>
> before that? (If it works, you have to make sure that other settings as 
> "ntsec" are retained in that environment variable, but not "nobinmode").
>
> Hth,
> Dscho
In a word, yes. I tried many things with mount and CYGWIN, none fixed 
the problem. What I don't have is a simple test case I can push upstream 
to demonstrate the failure mode. However, if this is a crlf issue I 
suspect this would not work in msys either, regardless of being able to 
find the issue in Cygwin.

Mark

Respun - Scripts to use bundles to move data between repos

[Mark Levedahl]

Respun patch for git-bundle / git-unbundle.

I am working a project using git where we have many repositories on
machines that can never be directly connected, but which need to have
the same objects and development history.  Existing git protocols offer
limited support: we can either a) publish and apply patch files branch
by branch, or b) copy an entire repository from one machine to another
and then do local push or fetch.  While both are workable, neither is a
completely satisfactory solution, so I wrote the attached scripts that
support a "bundle" transfer mechanism.  A bundle is a tar archive having
two files: a list of references as given by git-show-ref and a pack file
of objects from git-pack-objects.  git-bundle creates the bundle,
git-unbundle unpacks and applies at the receiving end.  The means of
transporting the bundle file between the machines is arbitrary (sneaker
net, email, etc all can work).

This transfer protocol leaves it to the user to assure that the objects
in the bundle are sufficient: git-fetch and git-push determine what is
needed by comparison, but no such conversation is possible here.  The
approach is to specificy a range of commits to include in the bundle:
e.g.  git-bundle master~10..master to get the last 10 commits.  Having
too many is fine: git at the receiving end happily ignores duplicates.
Having too few objects raises an error and references are updated only
if their full set of dependencies are found in the target repository.


---

Thanks to all for comments on my previous set of patches on this topic,
I have respun them, taking the following actions on the various
suggestions.

1) Reformat help and .txt to fit 80 column screen: Done.

2) Use tar for the bundle, not zip. Done.

3) Link into main makefile: Done (including rename of git* to git*.sh

4) Link into Documentation/cmd-list.perl: Done.

5) Source git-sh-setup: Done (including removal of --bare and git dir
   checks).

6) Use git-index-pack rather than git-unpack-objects -Done.

7) Pipe ref list directly into git-pack-objects. - doesn't handle
   rev-list-args such as --since=5.days.ago, piping through git-rev-list
   first does so I left it that way.

8) Shallow option was broken - removed as non-essential and too hard to
   fix. (can re-add properly later if someone really wants it).

9) Dont use git-fsck, instead tell receiver what commits are needed
   directly. Done, but with regrets. git-rev-list --objects-edge needs
   to be taught to work when given --max-age, git-bundle calculates the
   required result using a shell pipeline for the moment. Will revisit
   after re-educating git-rev-list. (git-bundle does use git-rev-list
   when --max-age is not present).

10) Allow --output file, rather than --output=file. Rejected as this
   required git-bundle to check arguments for git-rev-list rather than
   letting git-rev-parse do that, or requires a specific ordering of
   inputs. The first violates separation of functions (let git-rev-parse
   handle errors on arguments it takes), or simply trades one annoyance
   for another. No available solution is better than what is there now.

11) Don't use temporary files in git-bundle. Left as is. Shell scrambles
   binary data, so cannot be used to split out a binary pack file. This
   is the service tar provides, and tar will only build from existing
   files, so temporary files are needed unless this whole thing is
   re-written in C.


Mark Levedahl

[PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

Some workflows require coordinated development between repositories on
machines that can never be connected. This utility creates a bundle
containing a pack of objects and associated references (heads or tags)
that can be independently transferred to another machine, effectively
supporting git-push like operations between disconnected systems.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 git-bundle.sh |  103 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 103 insertions(+), 0 deletions(-)
 create mode 100755 git-bundle.sh

diff --git a/git-bundle.sh b/git-bundle.sh
new file mode 100755
index 0000000..c431b52
--- /dev/null
+++ b/git-bundle.sh
@@ -0,0 +1,103 @@
+#!/bin/sh
+# Create a bundle to carry from one git repo to another
+# (e.g., "sneaker-net" based push)
+# git-bundle <git-rev-list args>
+# git-bundle --bare <git-rev-list args>
+# creates bundle.tar in current directory (can rename of course)
+#
+# The bundle includes all refs given (--all selects every ref in the repo).
+# and all of the commit objects needed subject to the list given.
+#
+# Objects to be packed are limited by specifying one or more of
+#   ^commit-ish    - indicated commits already at the target
+#                    (can have more than one ^commit-ish)
+#   --since=xxx    - Assume target repo has all relevant commits
+#                    earlier than xxx
+
+USAGE='git-bundle [--output=file] <git-rev-list arguments>'
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+# pull out rev-list args vs program args, parse the latter
+gitrevargs=$(git-rev-parse --symbolic --revs-only $*) || exit 1
+myargs=$(git-rev-parse --no-revs $*) || exit 1
+
+bfile=bundle.tar
+nextisoutput=
+for arg in $myargs ; do
+	case "$arg" in
+		-h|--h|--he|--hel|--help)
+			echo "$USAGE"
+			exit;;
+		--output=*)
+			bfile=${arg##--output=};;
+		*)
+			die "unknown option: $arg";;
+	esac
+done
+
+# find the refs to carry along and get sha1s for each.
+refs=
+fullrevargs=
+for arg in $gitrevargs ; do
+	#ignore options and basis refs, get unambiguous ref name for things
+	# we will transport (e.g., user gives master, have heads/master and
+	# remotes/origin/master, we keep the former).
+	case "$arg" in
+		-*) fullrevargs="$fullrevargs $arg";;
+		^*) fullrevargs="$fullrevargs $arg";;
+		*)	ref=$(git-show-ref "$arg" | head -n1)
+			[ -z "$ref" ] && die "unknown reference: $arg"
+			fullrevargs="$fullrevargs ${ref#* }"
+			refs="$refs $ref"
+			;;
+	esac
+done
+[ -z "$refs" ] && die "No references specified, I don't know what to bundle."
+
+# git-rev-list cannot determine edge objects if a date restriction is given...
+# we do things a slow way if max-age or min-age are given
+fast=
+[ "${fullrevargs##*--max-age}" == "$fullrevargs" ] && \
+[ "${fullrevargs##*--min-age}" == "$fullrevargs" ] && fast=1
+
+if [ -z "$fast" ] ; then
+	# get a list of all commits that will be packed along with parents of each.
+	# A fixed git-rev-list --boundary should replace all of this.
+	echo "Finding prerequisites and commits to bundle..."
+	commits=$(git-rev-list $fullrevargs)
+	
+	# get immediate parents of each commit to include
+	parents=
+	for c in $commits ; do
+		parents="$parents $(git-rev-list --parents $c | head -1 | cut -b42-)"
+	done
+	parents=$(printf "%s\n" $parents | sort | uniq)
+	
+	# factor out what will be in this bundle, the remainder are the bundle's pre-requisites.
+	# double up commits in this as we only want things that are only in parents to appear once
+	prereqs=$(printf "%s\n" $parents $commits $commits | \
+		sort | \
+		uniq -c | \
+		grep ' 1 ' \
+		| sed 's/ *1 //')
+else
+	prereqs=$(git-rev-list --objects-edge $fullrevargs | \
+		grep '^-' | sed 's/-//')
+fi
+
+# create refs and pack
+[ -e "$bfile" ] && rm -f "$bfile" 2>/dev/null
+printf "%s\n" $prereqs > .gitBundleReferences
+echo "-" >> .gitBundleReferences
+git-show-ref $refs >> .gitBundleReferences
+(git-rev-list --objects $fullrevargs | \
+	cut -b-40 | \
+	git pack-objects --all-progress --stdout >.gitBundlePack) \
+	|| (rm -f "$bfile" ; exit)
+tar cf "$bfile"  .gitBundleReferences .gitBundlePack
+tar cf "$bfile"  .gitBundleReferences .gitBundlePack
+rm .gitBundleReferences .gitBundlePack
+
+# done
+echo "Created $bfile"
-- 
1.5.0.34.g6afaa

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

> +# find the refs to carry along and get sha1s for each.
> +refs=
> +fullrevargs=
> +for arg in $gitrevargs ; do
> +	#ignore options and basis refs, get unambiguous ref name for things
> +	# we will transport (e.g., user gives master, have heads/master and
> +	# remotes/origin/master, we keep the former).

I am not sure if that is a good behaviour.  What if user gives
xyzzy and there are tags/xyzzy and remotes/origin/xyzzy?

I suspect it would be much safer to error out in ambiguous cases.

> +# we do things a slow way if max-age or min-age are given
> +fast=
> +[ "${fullrevargs##*--max-age}" == "$fullrevargs" ] && \
> +[ "${fullrevargs##*--min-age}" == "$fullrevargs" ] && fast=1

Style.

Our scripts tend to spell the test command "test" and equality test
operator as single '='.  Also they tend to say:

        case "$fullrevargs" in
        *--max-age* | *--min-age*) ... 
        esac

> +	# get immediate parents of each commit to include
> +	parents=
> +	for c in $commits ; do
> +		parents="$parents $(git-rev-list --parents $c | head -1 | cut -b42-)"
> +	done

Nicely done, but you seemed to have used "head -n 1" elsewhere,
which is more portable.

> +	# factor out what will be in this bundle, the remainder are the bundle's pre-requisites.
> +	# double up commits in this as we only want things that are only in parents to appear once
> +	prereqs=$(printf "%s\n" $parents $commits $commits | \
> +		sort | \
> +		uniq -c | \
> +		grep ' 1 ' \
> +		| sed 's/ *1 //')

Don't pipe output of grep to sed ;-).

> +else
> +	prereqs=$(git-rev-list --objects-edge $fullrevargs | \
> +		grep '^-' | sed 's/-//')
> +fi

Likewise.  The latter is:

	... | sed -ne 's/^-//p'

> +# create refs and pack
> +[ -e "$bfile" ] && rm -f "$bfile" 2>/dev/null
> +printf "%s\n" $prereqs > .gitBundleReferences

Our scripts assume $GIT_DIR/ is writable but not necessarily the
working directory when they have to use temporary files.  Also
please clean temporaries even when you error out with traps.

	tmp=$GIT_DIR/bundle_tmp$$
	references="$tmp-references"
        pack="$tmp-pack"
        trap 'rm -f "$tmp-*"' 0 1 2 3 15

        echo "-" >>"$references" &&
        git-show-ref $refs >>"$references" &&
        (git-rev-list ... >"$pack") || exit

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Shawn O. Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Mark Levedahl <mdl123@verizon.net> writes:
> > +# we do things a slow way if max-age or min-age are given
> > +fast=
> > +[ "${fullrevargs##*--max-age}" == "$fullrevargs" ] && \
> > +[ "${fullrevargs##*--min-age}" == "$fullrevargs" ] && fast=1
> 
> Style.
> 
> Our scripts tend to spell the test command "test" and equality test
> operator as single '='.

Using "test" is a style thing, but using "=" rather than "==" is
a portability issue.  "==" is accepted by bash as a synonym for
"=", but its not valid elsewhere.

I've made my fair share of patches with "==" in them, only to have
someone else have a problem on their system and then submit a fix to
change them to "=".  Just say no to "==" in Git shell scripts.  ;-)
 
> > +	# get immediate parents of each commit to include
> > +	parents=
> > +	for c in $commits ; do
> > +		parents="$parents $(git-rev-list --parents $c | head -1 | cut -b42-)"
> > +	done
> 
> Nicely done, but you seemed to have used "head -n 1" elsewhere,
> which is more portable.

What about:

	parents="$parents $(git-rev-list --max-count=1 --parents $c | cut -b42-)"

?

I just used that trick recently.  Oh yea it was in git-gui's
GIT-VERSION-GEN script.  Though I did not know about using cut
like that...
 
-- 
Shawn.

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> Using "test" is a style thing, but using "=" rather than "==" is
> a portability issue.  "==" is accepted by bash as a synonym for
> "=", but its not valid elsewhere.

True, I should have stressed that.  Thanks.  Also in the same
sense (but to a lessor degree), "test" vs "[" is also
portability issue.

> What about:
>
> 	parents="$parents $(git-rev-list --max-count=1 --parents $c | cut -b42-)"
>
> I just used that trick recently.  Oh yea it was in git-gui's
> GIT-VERSION-GEN script.  Though I did not know about using cut
> like that...

I used to stay away from 'cut' myself, although it is not too
bad for portability these days.  I would probably have done the
above with a single sed process, though.

[PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

Some workflows require coordinated development between repositories on
machines that can never be connected. This utility creates a bundle
containing a pack of objects and associated references (heads or tags)
that can be independently transferred to another machine, effectively
supporting git-push like operations between disconnected systems.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 git-bundle.sh |  112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 112 insertions(+), 0 deletions(-)
 create mode 100755 git-bundle.sh

diff --git a/git-bundle.sh b/git-bundle.sh
new file mode 100755
index 0000000..f56f499
--- /dev/null
+++ b/git-bundle.sh
@@ -0,0 +1,112 @@
+#!/bin/sh
+# Create a bundle to carry from one git repo to another
+# (e.g., "sneaker-net" based push)
+# git-bundle <git-rev-list args>
+# git-bundle --bare <git-rev-list args>
+# creates bundle.tar in current directory (can rename of course)
+#
+# The bundle includes all refs given (--all selects every ref in the repo).
+# and all of the commit objects needed subject to the list given.
+#
+# Objects to be packed are limited by specifying one or more of
+#   ^commit-ish    - indicated commits already at the target
+#                    (can have more than one ^commit-ish)
+#   --since=xxx    - Assume target repo has all relevant commits
+#                    earlier than xxx
+
+USAGE='git-bundle [-o file | --output=file] <git-rev-list arguments>'
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+bfile=bundle.tar
+args=
+while test -n "$1" ; do
+    case $1 in
+        -h|--h|--he|--hel|--help)
+            echo "$USAGE"
+            exit;;
+        --output=*)
+            bfile=${1##--output=};;
+        -o|--output)
+            shift
+            bfile=$1;;
+        *)
+            args="$args $1"
+    esac
+    shift
+done
+
+unknown=$(git-rev-parse --no-revs $args)
+test -z "$unknown" || die "unknown option: $unknown"
+gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1
+
+# find the refs to carry along and get sha1s for each.
+refs=
+fullrevargs=
+for arg in $gitrevargs ; do
+    #ignore options and basis refs, get unambiguous ref name for things
+    # we will transport (e.g., user gives master, have heads/master and
+    # remotes/origin/master, we keep the former).
+    case "$arg" in
+        -*) fullrevargs="$fullrevargs $arg";;
+        ^*) fullrevargs="$fullrevargs $arg";;
+        *)  ref=$(git-show-ref "$arg")
+            test "$(echo $ref | wc -w)" = "2" || die "Ambigous reference: $arg
+$ref"
+            fullrevargs="$fullrevargs ${ref#* }"
+            refs="$refs $ref"
+            ;;
+    esac
+done
+test -z "$refs" && die "No references specified, I don't know what to bundle."
+
+# git-rev-list cannot determine edge objects if a date restriction is
+# given...  we do things a slow way if max-age or min-age are given
+case "$fullrevargs" in
+    *--max-age* | *--min-age*)
+    # get a list of all commits that will be packed along with
+    # parents of each.  A fixed git-rev-list --boundary should
+    # replace all of this.
+    echo "Finding prerequisites and commits to bundle..."
+    commits=$(git-rev-list $fullrevargs)
+
+    # get immediate parents of each commit to include
+    parents=
+    for c in $commits ; do
+        parents="$parents $(git-rev-list --parents --max-count=1 $c | cut -b42-)"
+    done
+    parents=$(printf "%s\n" $parents | sort | uniq)
+
+    # factor out what will be in this bundle, the remainder are the
+    # bundle's prerequisites.  double up commits in this as we only
+    # want things that are only in parents to appear once
+    prereqs=$(printf "%s\n" $parents $commits $commits | \
+        sort | uniq -c | sed -ne 's/^ *1 //p');;
+    *)
+        prereqs=$(git-rev-list --objects-edge $fullrevargs | sed -ne 's/^-//p');;
+esac
+
+# create refs and pack
+tmp=$GIT_DIR/bundle_tmp$$
+references="$tmp-references"
+pack="$tmp-pack"
+trap 'rm -f "$references" "$pack"' 0 1 2 3 15
+
+echo "v1" > "$references" &&
+echo "prerequisites" >> "$references" &&
+printf "%s\n" $prereqs >> "$references" &&
+echo "references" >> "$references" &&
+git-show-ref $refs >> "$references" &&
+(git-rev-list --objects $fullrevargs | \
+    cut -b-40 | \
+    git pack-objects --all-progress --stdout >> "$pack" ) \
+    || exit
+
+# create the tar file, clean up
+tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
+    --verbose --show-transformed-names \
+    "$references" "$pack"
+rm -f "$pack" "$references"
+
+# done
+echo "Created $bfile"
-- 
1.5.0.rc4.375.gd0938-dirty

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Shawn O. Pearce]

Mark Levedahl <mdl123@verizon.net> wrote:
> +# create the tar file, clean up
> +tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
> +    --verbose --show-transformed-names \
> +    "$references" "$pack"

I'm not sure this will work on FreeBSD.  Both 5.1 and 6.1 use tar
that does not know about --absolute-names, --transform, --verbose,
or --show-transformed-names.

-- 
Shawn.

Re: [PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Simon 'corecode' Schubert]

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

Shawn O. Pearce wrote:
> Mark Levedahl <mdl123@verizon.net> wrote:
>> +# create the tar file, clean up
>> +tar cf "$bfile" --absolute-names --transform="s,$tmp-,," \
>> +    --verbose --show-transformed-names \
>> +    "$references" "$pack"
> 
> I'm not sure this will work on FreeBSD.  Both 5.1 and 6.1 use tar
> that does not know about --absolute-names, --transform, --verbose,
> or --show-transformed-names.

for portability, pax is the official choice :)  and it can even do path name modifications with -s.  but why again are we using tar there?  this data could easiliy be put in one mixed text/binary file, starting out with

#!/bin/sh
echo "This is a git bundle.  Use git-unbundle to process me." >&1
exit
### DATA ###

or so

cheers
  simon

-- 
Serve - BSD     +++  RENT this banner advert  +++    ASCII Ribbon   /"\
Work - Mac      +++  space for low €€€ NOW!1  +++      Campaign     \ /
Party Enjoy Relax   |   http://dragonflybsd.org      Against  HTML   \
Dude 2c 2 the max   !   http://golden-apple.biz       Mail + News   / \


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]