Re: Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

Re: Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

>>
>Also as Shawn pointed out, the script too heavily depends on GNU
>tar.  Can we do something about it?

Let me ponder, I'm sure I can do something. As I noted in another response, Cygwin won't let me reliably pipe the pack file through bash, so I'm forced to use some archiver.

Mark

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

>>>
>>Also as Shawn pointed out, the script too heavily depends on GNU
>>tar.  Can we do something about it?
>
> Let me ponder, I'm sure I can do something. As I noted in
> another response, Cygwin won't let me reliably pipe the pack
> file through bash, so I'm forced to use some archiver.

Mark, how urgent do you want to have "bundle" in my tree?  As I
understand it, this came out of your working zip based
implementation your group already use, so I am suspecting that
you do not have urgent need to have a different one in git.git
in a half-baked shape.

The reason I say this is because I very much in favor of the
general idea of an archive file that can be used to sneakernet
repository updates, but I find the current bundle-unbundle
interface a bit awkward to integrate into the rest of the
system.

Wouldn't it be nice if you can treat a bundle as just a
different kind of git URL that you can "git fetch"?

	$ git fetch file.bdl 'refs/heads/*:refs/heads/*'

As the way I read the current implementation of git-unbundle is
that it is designed to only overwrite (with fast-forward check
not to lose changes from the target) all the refs in place.  It
does not, for example, allow extracting only one branch, nor
storing the branch in a tracking branch by renaming.  I think it
would be nicer to change the external interface from the bundle
subsystem to the calling scripts, so that it behaves closer to
git-fetch-pack.

I haven't thought things through, but I think something like this
would be sufficient:

	# create a bundle.
	$ git bundle --create v1.5.0..maint master next >file.bdl

	# you can view the tips of refs it contains.  This is
        # similar to "ls-remote" output.
	$ git bundle --list-heads file.bdl
        efa13f7b7ea1605deab3a6478fa0d0706c828170 refs/heads/maint
        af99711cd84c30a16450f73dbc21ba9f9f9803e6 refs/heads/master
        664e83a22b604fc5af1a84ddd48549b005cf4bc9 refs/heads/next

        # optionally allow checking if the pre-requisites of the
        # bundle are available.
	$ git bundle --verify file.bdl
	error: file.bdl depends on the following commits you lack
	in the repository:
        82bf92f9846326a743102e27fa9827422dddfada v1.5.0-564-g82bf92f
	4d462883de41190afd23672c8236361c61a9e6bd v1.5.0-34-g4d46288

	# extract the packfile part, run index-pack and report
        # the result the same way as git-fetch-pack
        $ git bundle --unbundle file.bdl
	pack	340e3faa26616ca8c38b369a6d323ecf7a34f4fb
        efa13f7b7ea1605deab3a6478fa0d0706c828170 refs/heads/maint
        af99711cd84c30a16450f73dbc21ba9f9f9803e6 refs/heads/master
        664e83a22b604fc5af1a84ddd48549b005cf4bc9 refs/heads/next

Then git-ls-remote can be taught about a bundle file and use the
'git bundle --list-heads'.  Also, with something like this in
your config:

	[remote "bundle"]
        	url = /home/me/tmp/file.bdl
                fetch = refs/heads/*:refs/remotes/origin/*

You can first sneakernet the bundle file to ~/tmp/file.bdl and
then these commands:

	$ git ls-remote bundle
        $ git fetch bundle
	$ git pull bundle

would treat it as if it is talking with a remote side over the
network.

Hmm?

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

Junio C Hamano wrote:
> Mark, how urgent do you want to have "bundle" in my tree?  As I
> understand it, this came out of your working zip based
> implementation your group already use, so I am suspecting that
> you do not have urgent need to have a different one in git.git
> in a half-baked shape.
>   
I can get my work done today with what I have. However, I really believe 
bundle is a good addition to git, and I don't want to maintain an 
out-of-tree patch to git to add this capability.
> Wouldn't it be nice if you can treat a bundle as just a
> different kind of git URL that you can "git fetch"?
>
> 	$ git fetch file.bdl 'refs/heads/*:refs/heads/*'
>   
yes
>
> Then git-ls-remote can be taught about a bundle file and use the
> 'git bundle --list-heads'.  Also, with something like this in
> your config:
>
> 	[remote "bundle"]
>         	url = /home/me/tmp/file.bdl
>                 fetch = refs/heads/*:refs/remotes/origin/*
>
> You can first sneakernet the bundle file to ~/tmp/file.bdl and
> then these commands:
>
> 	$ git ls-remote bundle
>         $ git fetch bundle
> 	$ git pull bundle
>
> would treat it as if it is talking with a remote side over the
> network.
>
> Hmm?
>
>   
As long as I can still do a "git fetch file.bdl" and without having to 
do the config stuff. I'm happy. Integrating this bundle with basic git 
approaches to things is obviously good. The frontend you outlined seems 
a trivial rearrangement of what I already have, but I'll let this 
discussion progress a bit further before I start doing that.

Mark

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

>> Wouldn't it be nice if you can treat a bundle as just a
>> different kind of git URL that you can "git fetch"?
>>
>> 	$ git fetch file.bdl 'refs/heads/*:refs/heads/*'
>>
> yes
>...
>
> As long as I can still do a "git fetch file.bdl" and without having to
> do the config stuff. I'm happy.

I do not think your patch (original or respun) checks if it is
overwriting the current branch.  Even if it is a fast forward,
it should check this condition and prevent the end user from
getting confused.  The above sample command line you quoted from
my message can potentially have the same problem, but "git
fetch" checks and refuses.

Even when the "remote" is not a bundle file, it sometimes is
useful to store the refs as they are without mapping them to
remotes hierarchy as the separate-remote layout does.  An
obvious use case is to mirror another repository as is.  So it
may make sense to teach --mirror to "git-fetch" so that you can
say:

	$ git fetch --mirror file.bdl

which would be a short-hand to say:

	$ git fetch file.bdl 'refs/*:refs/*'

A final note.  A real 'mirror' mode should also remove stale
refs that do not exist on the remote side anymore, which is a
different use case as your bundle, which presumably is primarily
meant to carry not all but only selected set of refs, and most
likely not the 'master' branch head (and I am guessing that that
is why you forgot to make sure you are not overwriting the
current branch in the unbundle script).  A real 'mirror' mode
would use a separate option to remove a ref that does not exist
on the remote end anymore, like:

	$ git fetch --mirror-all git://git.kernel.org/pub/scm/git/git.git/

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

Junio C Hamano wrote:
> I do not think your patch (original or respun) checks if it is
> overwriting the current branch.  Even if it is a fast forward,
> it should check this condition and prevent the end user from
> getting confused.  The above sample command line you quoted from
> my message can potentially have the same problem, but "git
> fetch" checks and refuses.
>   
True. I've been using this to mirror remote/* and tags. Making this just 
an alternate transport behind fetch and pull is clearly the cleanest way 
to deal with non-remote branches.
> A final note.  A real 'mirror' mode should also remove stale
> refs that do not exist on the remote side anymore, which is a
> different use case as your bundle, which presumably is primarily
> meant to carry not all but only selected set of refs, and most
> likely not the 'master' branch head (and I am guessing that that
> is why you forgot to make sure you are not overwriting the
> current branch in the unbundle script).  A real 'mirror' mode
> would use a separate option to remove a ref that does not exist
> on the remote end anymore, like:
>
> 	$ git fetch --mirror-all git://git.kernel.org/pub/scm/git/git.git
>   
Perhaps "git fetch --mirror --delete" would be more suggestive of the 
difference to "git fetch --mirror"?

Mark

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

> Junio C Hamano wrote:
> ...
>> A final note.  A real 'mirror' mode should also remove stale
>> refs that do not exist on the remote side anymore, which is a
>> different use case as your bundle,...
>> ...  A real 'mirror' mode
>> would use a separate option to remove a ref that does not exist
>> on the remote end anymore, like:
>>
>> 	$ git fetch --mirror-all git://git.kernel.org/pub/scm/git/git.git
>>
> Perhaps "git fetch --mirror --delete" would be more suggestive of the
> difference to "git fetch --mirror"?

I think that is a good suggestion, as people familiar with rsync
already know what --delete means in such a context.

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

Junio C Hamano wrote:
> I haven't thought things through, but I think something like this
> would be sufficient:
>
> 	# create a bundle.
> 	$ git bundle --create v1.5.0..maint master next >file.bdl
>
>   
In this vein, would it make sense to let git-push be the front end to 
create the bundle? I'm not sure git-push really contributes anything, 
but the interface would then be consistent across all transports. Just a 
thought.

Mark

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

> Junio C Hamano wrote:
>> I haven't thought things through, but I think something like this
>> would be sufficient:
>>
>> 	# create a bundle.
>> 	$ git bundle --create v1.5.0..maint master next >file.bdl
>>
>>
> In this vein, would it make sense to let git-push be the front end to
> create the bundle? I'm not sure git-push really contributes anything,
> but the interface would then be consistent across all transports. Just
> a thought.

I am not sure.  For the "fetch" case, the UI will be exactly the
same (URL and then refspecs), but we never say bottom commits
for git-push.  When you are creating a bundle, unless you are
including everything, you want a way to say which bottom commits
to be excluded from the resulting pack, so they are different.
I had an impression bundle creation is similar to archive.

[PATCH] Add git-bundle: move objects and references by archive.

[Mark Levedahl]

Some workflows require use of repositories on machines that cannot be
connected, preventing use of git-fetch / git-push to transport objects
and references between the repositories.

git-bundle provides an alternate transport mechanism, effectively allowing
git-fetch and git-pull to operate using sneakernet transport. git-bundle
--create allows the user to create a bundle containing one or more branches
or tags, but with specified basis assumed to exist on the target repository.
At the receiving end, git-bundle acts like git-fetch-pack, allowing the
user to invoke git-fetch or git-pull using the bundle file as the URL.
git-fetch and git-ls-remote determine they have a bundle URL by checking
that the URL points to a file, but are otherwise unchanged in operation
with bundles.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 Documentation/cmd-list.perl  |    1 +
 Documentation/git-bundle.txt |  146 +++++++++++++++++++++++++++++++++++
 Makefile                     |    3 +-
 git-bundle.sh                |  174 ++++++++++++++++++++++++++++++++++++++++++
 git-fetch.sh                 |   11 ++-
 git-ls-remote.sh             |    7 ++-
 6 files changed, 338 insertions(+), 4 deletions(-)
 mode change 100755 => 100644 Documentation/cmd-list.perl
 create mode 100644 Documentation/git-bundle.txt
 create mode 100755 git-bundle.sh

diff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl
old mode 100755
new mode 100644
index a2d6268..f61c77a
--- a/Documentation/cmd-list.perl
+++ b/Documentation/cmd-list.perl
@@ -70,6 +70,7 @@ git-archive                             mainporcelain
 git-bisect                              mainporcelain
 git-blame                               ancillaryinterrogators
 git-branch                              mainporcelain
+git-bundle                              mainporcelain
 git-cat-file                            plumbinginterrogators
 git-checkout-index                      plumbingmanipulators
 git-checkout                            mainporcelain
diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt
new file mode 100644
index 0000000..27db785
--- /dev/null
+++ b/Documentation/git-bundle.txt
@@ -0,0 +1,146 @@
+git-bundle(1)
+=============
+
+NAME
+----
+git-bundle - Move objects and refs by archive
+
+
+SYNOPSIS
+--------
+'git-bundle' --create file <git-rev-list args> <--tar tarspec>
+'git-bundle' --verify file  <--tar tarspec>
+'git-bundle' --list-heads file <reflist> <--tar tarspec>
+'git-bundle' --unbundle file <reflist> <--tar tarspec>
+
+DESCRIPTION
+-----------
+
+Some workflows require that one or more branches of development on one
+machine be replicated on another machine, but the two machines cannot
+be directly connected so the interactive git protocols (git, ssh,
+rsync, http) cannot be used.  This command provides suport for
+git-fetch and git-pull to operate by packaging objects and references
+in an archive at the originating machine, then importing those into
+another repository using gitlink:git-fetch[1] and gitlink:git-pull[1]
+after moving the archive by some means (i.e., by sneakernet).  As no
+direct connection between repositories exists, the user must specify a
+basis for the bundle that is held by the destination repository: the
+bundle assumes that all objects in the basis are already in the
+destination repository.
+
+OPTIONS
+-------
+
+--create file::
+       Used to create a bundle named 'file'.  This requires the
+       git-rev-list arguments to define the bundle contents.
+
+--verify file::
+       Used to check that a bundle file is valid and will apply
+       cleanly to the current repository.  This includes checks on the
+       bundle format itself as well as checking that the prerequisite
+       commits exist and are fully linked in the current repository.
+       git-bundle prints a list of missing commits, if any, and exits
+       with non-zero status.
+
+--list-heads file::
+       Lists the references defined in the bundle.  If followed by a
+       list of references, only references matching those given are
+       printed out.
+
+--unbundle file::
+       Passes the objects in the bundle to gitlink:git-index-pack[1]
+       for storage in the repository, then prints the names of all
+       defined references. If a reflist is given, only references
+       matching those in the given list are printed. This command is
+       really plumbing, intended to be called only by
+       gitlink:git-fetch[1].
+
+git-rev-list args::
+       A list of arguments, accepatble to git-rev-parse and
+       git-rev-list, that specify the specific objects and references
+       to transport.  For example, "master~10..master" causes the
+       current master reference to be packaged along with all objects
+       added since its 10th ancestor commit.  There is no explicit
+       limit to the number of references and objects that may be
+       packaged.
+
+
+reflist::
+       A list of references used to limit the references reported as
+       available. This is principally of use to git-fetch, which
+       expects to recieve only those references asked for and not
+       necessarily everything in the pack (in this case, git-bundle is
+       acting like gitlink:git-fetch-pack[1]).
+
+tar tarspec::
+
+       git-bundle uses tar, and requires a tar supporting c, f, and
+       -O. By default, git-bundle looks for gtar on the path, then for
+       tar if gtar is not found. This can be overridden by explicitly
+       defining tarspec, or by defining TAR in the environment.
+
+SPECIFYING REFERENCES
+---------------------
+
+git-bundle will only package references that are shown by
+git-show-ref: this includes heads, tags, and remote heads.  References
+such as master~1 cannot be packaged, but are perfectly suitable for
+defining the basis.  More than one reference may be packaged, and more
+than one basis can be specified.  The objects packaged are those not
+contained in the union of the given bases.  Each basis can be
+specified explicitly (e.g., ^master~10), or implicitly (e.g.,
+master~10..master, master --since=10.days.ago).
+
+It is very important that the basis used be held by the destination.
+It is ok to err on the side of conservatism, causing the bundle file
+to contain objects already in the destination as these are ignored
+when unpacking at the destination.
+
+EXAMPLE
+-------
+
+Assume two repositories exist as R1 on machine A, and R2 on machine B.
+For whatever reason, direct connection between A and B is not allowed,
+but we can move data from A to B via some mechanism (CD, email, etc).
+We want to update R2 with developments made on branch master in R1.
+We set a tag in R1 (lastR2bundle) after the previous such transport,
+and move it afterwards to help build the bundle.
+
+in R1 on A:
+$ git-bundle --create mybundle master ^lastR2bundle
+$ git tag -f lastR2bundle master
+
+(move mybundle from A to B by some mechanism)
+
+in R2 on B:
+$ git-bundle --verify mybundle
+$ git-fetch mybundle  refspec
+
+where refspec is refInBundle:localRef
+
+
+Also, with something like this in your config:
+
+[remote "bundle"]
+    url = /home/me/tmp/file.bdl
+    fetch = refs/heads/*:refs/remotes/origin/*
+
+You can first sneakernet the bundle file to ~/tmp/file.bdl and
+then these commands:
+
+$ git ls-remote bundle
+$ git fetch bundle
+$ git pull bundle
+
+would treat it as if it is talking with a remote side over the
+network.
+
+Author
+------
+Written by Mark Levedahl <mdl123@verizon.net>
+
+GIT
+---
+Part of the gitlink:git[7] suite
diff --git a/Makefile b/Makefile
index ebecbbd..c6d540e 100644
--- a/Makefile
+++ b/Makefile
@@ -177,7 +177,8 @@ SCRIPT_SH = \
 	git-applymbox.sh git-applypatch.sh git-am.sh \
 	git-merge.sh git-merge-stupid.sh git-merge-octopus.sh \
 	git-merge-resolve.sh git-merge-ours.sh \
-	git-lost-found.sh git-quiltimport.sh
+	git-lost-found.sh git-quiltimport.sh \
+	git-bundle.sh
 
 SCRIPT_PERL = \
 	git-add--interactive.perl \
diff --git a/git-bundle.sh b/git-bundle.sh
new file mode 100755
index 0000000..2bb9232
--- /dev/null
+++ b/git-bundle.sh
@@ -0,0 +1,174 @@
+#!/bin/sh
+# Basic handler for bundle files to connect repositories via sneakernet.
+# Invocation must include action.
+# This function can create a bundle or provide information on an existing bundle
+# supporting git-fetch, git-pull, and git-ls-remote
+
+USAGE='[--create bundle <git-rev-list args>] |
+[--verify|--list-heads|--unbundle bundle] <--tar tarspec>
+     where bundle is the name of the bundle file.'
+
+verify() {
+    # Check bundle version
+    test -r "$bfile" || die "cannot find $bfile"
+    test "$($TAR -xf ""$bfile"" -O version)" = "v1 git-bundle" ||
+        die "$bfile doesn't look like a v1 bundle file."
+
+	# do fast check, then if any prereqs are missing then go line by line
+	# to be verbose about the errors
+    prereqs=$($TAR xf "$bfile" -O prerequisites)
+	test -z "$prereqs" && return 0
+	bad=$(echo "$prereqs" | cut -b-40 | git-rev-list --stdin --not --all 2>&1)
+    if test -n "$bad" ; then
+		test "$1" = "--silent" && return 1
+        echo "error: $bfile requires the following commits you lack:"
+		echo "$prereqs" |
+		while read sha1 comment ; do
+            missing=$(git-rev-list $sha1 --not --all 2>&1)
+            test -n "$missing" && echo "$sha1 $comment"
+		done
+        exit 1
+    fi
+    return 0
+}
+
+# list all or just a subset
+list_heads() {
+	if test -z "$*" ; then
+		$TAR -xf "$bfile" -O references 2>/dev/null || exit 1
+	else
+		($TAR -xf "$bfile" -O references 2>/dev/null || exit 1) |
+		while read sha1 ref ; do
+			for arg in $* ; do
+				if test "${ref%$arg}" != "$ref" ; then
+					echo "$sha1 $ref"
+					break
+				fi
+			done
+		done
+	fi
+}
+
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+args=
+action=
+while test -n "$1" ; do
+    case $1 in
+        --create|--list-heads|--unbundle|--verify)
+            action=${1#--}
+            shift
+            bfile=$1
+            test -z "$bfile" && die "$action requires filename";;
+        --tar=*)
+            TAR=${1##--tar=};;
+        --tar)
+			shift
+            TAR=$1;;
+        *)
+            args="$args $1";;
+    esac
+    shift
+done
+test -z "$action" && die "No action given, what should I do?"
+
+# what tar to use, prefer gtar, then tar.
+if test -z "$TAR" ; then
+    GTAR=$(which gtar 2>/dev/null)
+    TAR=${GTAR:-tar}
+fi
+
+case $action in
+create)
+    unknown=$(git-rev-parse --no-revs $args)
+    test -z "$unknown" || die "unknown option: $unknown"
+    gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1
+
+    # find the refs to carry along and get sha1s for each.
+    refs=
+    fullrevargs=
+    for arg in $gitrevargs ; do
+        #ignore options and basis refs, get full ref name for things
+		# we will transport rejecting anything ambiguous (e.g., user
+		# gives master, have heads/master and remotes/origin/master, we
+		# keep the former).
+        case "$arg" in
+            -* | ^*) fullrevargs="$fullrevargs $arg";;
+            *)  ref=$(git-show-ref "$arg")
+                test "$(echo $ref | wc -w)" = "2" || die "Ambigous reference: $arg
+$ref"
+                fullrevargs="$fullrevargs ${ref#* }"
+                refs="$refs $ref";;
+        esac
+    done
+    test -z "$refs" && die "No references specified, I don't know what to bundle."
+
+    # git-rev-list cannot determine edge objects if a date restriction is
+    # given...  we do things a slow way if max-age or min-age are given
+    case "$fullrevargs" in
+        *--max-age* | *--min-age*)
+            # get a list of all commits that will be packed along with
+            # parents of each.  A fixed git-rev-list --boundary should
+            # replace all of this.
+            echo "Finding prerequisites and commits to bundle..."
+            commits=$(git-rev-list $fullrevargs)
+
+            # get immediate parents of each commit to include
+            parents=
+            for c in $commits ; do
+                parents="$parents $(git-rev-list --parents --max-count=1 $c | cut -b42-)"
+            done
+            parents=$(printf "%s\n" $parents | sort | uniq)
+
+            # factor out what will be in this bundle, the remainder are the
+            # bundle's prerequisites.  double up commits in this as we only
+            # want things that are only in parents to appear once
+            prereqs=$(printf "%s\n" $parents $commits $commits | \
+                sort | uniq -c | sed -ne 's/^ *1 //p');;
+        *)
+            prereqs=$(git-rev-list --objects-edge $fullrevargs | sed -ne 's/^-//p');;
+    esac
+
+	# replace prereqs with annotated version of same
+
+    # create refs and pack
+    tmp="$GIT_DIR/bundle_tmp$$"
+    prerequisites="$tmp/prerequisites"
+    references="$tmp/references"
+    version="$tmp/version"
+    pack="$tmp/pack"
+    trap 'rm -rf "$tmp"' 0 1 2 3 15
+
+    mkdir "$tmp" &&
+    echo "v1 git-bundle" > "$version" &&
+	touch "$prerequisites" &&
+	(for p in $prereqs ; do
+		git-rev-list --pretty=one --max-count=1 $p
+	done) > "$prerequisites" &&
+    git-show-ref $refs > "$references" &&
+    git-rev-list --objects $fullrevargs | cut -b-40 |
+        git pack-objects --all-progress --stdout > "$pack" &&
+
+    # create the tar file, clean up
+    cd "$tmp" &&
+    tar cf bundle prerequisites references version pack &&
+    cd - &&
+    mv "$tmp/bundle" "$bfile" &&
+    rm -rf "$tmp"
+
+    # done
+    echo "Created $bfile";;
+
+verify)
+    verify && echo "$bfile is ok";;
+
+list-heads)
+    list_heads $args;;
+
+unbundle)
+    verify --silent || exit 1
+    $TAR -xf "$bfile" -O pack | git-index-pack --stdin ||
+        die "error: $bfile has a corrupted pack file"
+    list_heads $args;;
+esac
diff --git a/git-fetch.sh b/git-fetch.sh
index ca984e7..4cb4009 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -377,8 +377,15 @@ fetch_main () {
     ( : subshell because we muck with IFS
       IFS=" 	$LF"
       (
-	  git-fetch-pack --thin $exec $keep $shallow_depth "$remote" $rref ||
-	  echo failed "$remote"
+	if test -r "$remote" ; then
+	    test -n "$shallow_depth" &&
+		die "shallow clone with bundle is not supported"
+	    git-bundle --unbundle "$remote" $rref ||
+	    echo failed "$remote"
+	else
+	    git-fetch-pack --thin $exec $keep $shallow_depth "$remote" $rref ||
+	    echo failed "$remote"
+	fi
       ) |
       (
 	trap '
diff --git a/git-ls-remote.sh b/git-ls-remote.sh
index 8ea5c5e..73b745d 100755
--- a/git-ls-remote.sh
+++ b/git-ls-remote.sh
@@ -89,8 +89,13 @@ rsync://* )
 	;;
 
 * )
-	git-peek-remote $exec "$peek_repo" ||
+	if test -r "$peek_repo" ; then
+		git bundle --list-heads "$peek_repo" ||
 		echo "failed	slurping"
+	else
+		git-peek-remote $exec "$peek_repo" ||
+		echo "failed	slurping"
+	fi
 	;;
 esac |
 sort -t '	' -k 2 |
-- 
1.5.0.rc4.375.gd0938-dirty

Re: [PATCH] Add git-bundle: move objects and references by archive.

[Mark Levedahl]

Mark Levedahl wrote:
> -	git-peek-remote $exec "$peek_repo" ||
> +	if test -r "$peek_repo" ; then
>   
oops:                   test    -f
actually works. Corrected patch follows.

Mark

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

Mark Levedahl wrote:
>> Also as Shawn pointed out, the script too heavily depends on GNU
>> tar.  Can we do something about it
>>     
This one is easy, make tmp a directory, then build the tar file in that 
directory so the archive members don't include the tmp name, then just 
move the tar file to where it is needed...

tmp="$GIT_DIR/bundle_tmp$$"
references="$tmp/references"
pack="$tmp/pack"
trap 'rm -rf "$tmp"' 0 1 2 3 15

mkdir "$tmp" &&
echo "v1 git-bundle" > "$version" &&
------

# create the tar file, clean up
cd "$tmp" &&
tar cf bundle prerequisites references version pack &&
cd - &&
mv "$tmp/bundle" "$bfile" &&
rm -rf "$tmp"

I believe that as this works, tar needs only to understand c, f, and -O. 
It is easy enough to search for gtar and use that, or allow user to 
define TAR. So, I think this is a non-issue.

Mark

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Johannes Schindelin]

Hi,

On Fri, 16 Feb 2007, Mark Levedahl wrote:

> Mark Levedahl wrote:
> > > Also as Shawn pointed out, the script too heavily depends on GNU 
> > > tar.  Can we do something about it
> > >     
> This one is easy, make tmp a directory, then build the tar file in that
> directory so the archive members don't include the tmp name, then just move
> the tar file to where it is needed...

Why not just get rid of this unneeded dependency? It is _unneeded_. Maybe 
you never suffered dependency hell (this is the only way I can explain 
your resistance). It is _good_ to get rid of dependencies.

Ah whatever, if you don't just get rid of that tar dependency, I'll just 
do it.

Ciao,
Dscho

Respun - Scripts to use bundles to move data between repos

[Mark Levedahl]

Respun patch for git-bundle / git-unbundle.

I am working a project using git where we have many repositories on
machines that can never be directly connected, but which need to have
the same objects and development history.  Existing git protocols offer
limited support: we can either a) publish and apply patch files branch
by branch, or b) copy an entire repository from one machine to another
and then do local push or fetch.  While both are workable, neither is a
completely satisfactory solution, so I wrote the attached scripts that
support a "bundle" transfer mechanism.  A bundle is a tar archive having
two files: a list of references as given by git-show-ref and a pack file
of objects from git-pack-objects.  git-bundle creates the bundle,
git-unbundle unpacks and applies at the receiving end.  The means of
transporting the bundle file between the machines is arbitrary (sneaker
net, email, etc all can work).

This transfer protocol leaves it to the user to assure that the objects
in the bundle are sufficient: git-fetch and git-push determine what is
needed by comparison, but no such conversation is possible here.  The
approach is to specificy a range of commits to include in the bundle:
e.g.  git-bundle master~10..master to get the last 10 commits.  Having
too many is fine: git at the receiving end happily ignores duplicates.
Having too few objects raises an error and references are updated only
if their full set of dependencies are found in the target repository.


---

Thanks to all for comments on my previous set of patches on this topic,
I have respun them, taking the following actions on the various
suggestions.

1) Reformat help and .txt to fit 80 column screen: Done.

2) Use tar for the bundle, not zip. Done.

3) Link into main makefile: Done (including rename of git* to git*.sh

4) Link into Documentation/cmd-list.perl: Done.

5) Source git-sh-setup: Done (including removal of --bare and git dir
   checks).

6) Use git-index-pack rather than git-unpack-objects -Done.

7) Pipe ref list directly into git-pack-objects. - doesn't handle
   rev-list-args such as --since=5.days.ago, piping through git-rev-list
   first does so I left it that way.

8) Shallow option was broken - removed as non-essential and too hard to
   fix. (can re-add properly later if someone really wants it).

9) Dont use git-fsck, instead tell receiver what commits are needed
   directly. Done, but with regrets. git-rev-list --objects-edge needs
   to be taught to work when given --max-age, git-bundle calculates the
   required result using a shell pipeline for the moment. Will revisit
   after re-educating git-rev-list. (git-bundle does use git-rev-list
   when --max-age is not present).

10) Allow --output file, rather than --output=file. Rejected as this
   required git-bundle to check arguments for git-rev-list rather than
   letting git-rev-parse do that, or requires a specific ordering of
   inputs. The first violates separation of functions (let git-rev-parse
   handle errors on arguments it takes), or simply trades one annoyance
   for another. No available solution is better than what is there now.

11) Don't use temporary files in git-bundle. Left as is. Shell scrambles
   binary data, so cannot be used to split out a binary pack file. This
   is the service tar provides, and tar will only build from existing
   files, so temporary files are needed unless this whole thing is
   re-written in C.


Mark Levedahl

[PATCH] Add git-bundle - pack objects and references for disconnected transfer

[Mark Levedahl]

Some workflows require coordinated development between repositories on
machines that can never be connected. This utility creates a bundle
containing a pack of objects and associated references (heads or tags)
that can be independently transferred to another machine, effectively
supporting git-push like operations between disconnected systems.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 git-bundle.sh |  103 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 103 insertions(+), 0 deletions(-)
 create mode 100755 git-bundle.sh

diff --git a/git-bundle.sh b/git-bundle.sh
new file mode 100755
index 0000000..c431b52
--- /dev/null
+++ b/git-bundle.sh
@@ -0,0 +1,103 @@
+#!/bin/sh
+# Create a bundle to carry from one git repo to another
+# (e.g., "sneaker-net" based push)
+# git-bundle <git-rev-list args>
+# git-bundle --bare <git-rev-list args>
+# creates bundle.tar in current directory (can rename of course)
+#
+# The bundle includes all refs given (--all selects every ref in the repo).
+# and all of the commit objects needed subject to the list given.
+#
+# Objects to be packed are limited by specifying one or more of
+#   ^commit-ish    - indicated commits already at the target
+#                    (can have more than one ^commit-ish)
+#   --since=xxx    - Assume target repo has all relevant commits
+#                    earlier than xxx
+
+USAGE='git-bundle [--output=file] <git-rev-list arguments>'
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+# pull out rev-list args vs program args, parse the latter
+gitrevargs=$(git-rev-parse --symbolic --revs-only $*) || exit 1
+myargs=$(git-rev-parse --no-revs $*) || exit 1
+
+bfile=bundle.tar
+nextisoutput=
+for arg in $myargs ; do
+	case "$arg" in
+		-h|--h|--he|--hel|--help)
+			echo "$USAGE"
+			exit;;
+		--output=*)
+			bfile=${arg##--output=};;
+		*)
+			die "unknown option: $arg";;
+	esac
+done
+
+# find the refs to carry along and get sha1s for each.
+refs=
+fullrevargs=
+for arg in $gitrevargs ; do
+	#ignore options and basis refs, get unambiguous ref name for things
+	# we will transport (e.g., user gives master, have heads/master and
+	# remotes/origin/master, we keep the former).
+	case "$arg" in
+		-*) fullrevargs="$fullrevargs $arg";;
+		^*) fullrevargs="$fullrevargs $arg";;
+		*)	ref=$(git-show-ref "$arg" | head -n1)
+			[ -z "$ref" ] && die "unknown reference: $arg"
+			fullrevargs="$fullrevargs ${ref#* }"
+			refs="$refs $ref"
+			;;
+	esac
+done
+[ -z "$refs" ] && die "No references specified, I don't know what to bundle."
+
+# git-rev-list cannot determine edge objects if a date restriction is given...
+# we do things a slow way if max-age or min-age are given
+fast=
+[ "${fullrevargs##*--max-age}" == "$fullrevargs" ] && \
+[ "${fullrevargs##*--min-age}" == "$fullrevargs" ] && fast=1
+
+if [ -z "$fast" ] ; then
+	# get a list of all commits that will be packed along with parents of each.
+	# A fixed git-rev-list --boundary should replace all of this.
+	echo "Finding prerequisites and commits to bundle..."
+	commits=$(git-rev-list $fullrevargs)
+	
+	# get immediate parents of each commit to include
+	parents=
+	for c in $commits ; do
+		parents="$parents $(git-rev-list --parents $c | head -1 | cut -b42-)"
+	done
+	parents=$(printf "%s\n" $parents | sort | uniq)
+	
+	# factor out what will be in this bundle, the remainder are the bundle's pre-requisites.
+	# double up commits in this as we only want things that are only in parents to appear once
+	prereqs=$(printf "%s\n" $parents $commits $commits | \
+		sort | \
+		uniq -c | \
+		grep ' 1 ' \
+		| sed 's/ *1 //')
+else
+	prereqs=$(git-rev-list --objects-edge $fullrevargs | \
+		grep '^-' | sed 's/-//')
+fi
+
+# create refs and pack
+[ -e "$bfile" ] && rm -f "$bfile" 2>/dev/null
+printf "%s\n" $prereqs > .gitBundleReferences
+echo "-" >> .gitBundleReferences
+git-show-ref $refs >> .gitBundleReferences
+(git-rev-list --objects $fullrevargs | \
+	cut -b-40 | \
+	git pack-objects --all-progress --stdout >.gitBundlePack) \
+	|| (rm -f "$bfile" ; exit)
+tar cf "$bfile"  .gitBundleReferences .gitBundlePack
+tar cf "$bfile"  .gitBundleReferences .gitBundlePack
+rm .gitBundleReferences .gitBundlePack
+
+# done
+echo "Created $bfile"
-- 
1.5.0.34.g6afaa

[PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

Some workflows require coordinated development between repositories on
machines that can never be connected. This utility unpacks a bundle
containing objects and associated references (heads or tags) into the
current repository, effectively supporting git-push like operations
between disconnected systems.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 git-unbundle.sh |   67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 67 insertions(+), 0 deletions(-)
 create mode 100755 git-unbundle.sh

diff --git a/git-unbundle.sh b/git-unbundle.sh
new file mode 100755
index 0000000..ca13305
--- /dev/null
+++ b/git-unbundle.sh
@@ -0,0 +1,67 @@
+#!/bin/sh
+# unpack a git-bundle file into current repository
+#
+# See git-bundle.
+
+USAGE="usage: git-unbundle [-f|--force] [file]"
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+bfile=bundle.tar
+force=
+while case "$#" in 0) break ;; esac
+do
+	case "$1" in
+	-f|--f|--fo|--for|--forc|--force)
+		force=1;;
+	-h|--h|--he|--hel|--help)
+		usage;;
+	-*)
+		die "unknown option: $1";;
+	*)
+		bfile="$1";;
+	esac
+	shift
+done
+
+[ -e "$bfile" ] || die "cannot find $bfile"
+
+# Get prereqs and refs in one go to avoid a third tar invocation.
+# Prereqs come first, are separated by a single - from the actual refs.
+refdata=$(tar -f "$bfile" -x --to-stdout  .gitBundleReferences)
+prereqs="${refdata%%-*}"
+refs="${refdata#*-}"
+
+# check that the prerequisites exist before unbundling.
+for sha1 in $prereqs ; do
+	git-rev-parse --verify $sha1 >& /dev/null || die "Prerequisite commit $sha1 not found."
+done
+
+# get the objects, maybe repack
+tar -f "$bfile" -x --to-stdout .gitBundlePack | git-index-pack --stdin
+
+# check each reference, assure that the result would be valid before updating local ref
+printf "%s %s\n" $refs | while read sha1 ref ; do
+	ok=
+	if [ -z "$force" ] ; then
+		# update only if non-fastforward
+		local=$(git-rev-parse --verify "$ref^0" 2>/dev/null)
+		if [ ! -z "$local" ] ; then
+			mb=$(git-merge-base $local $sha1)
+			if [ "$mb" != "$local" ] ; then
+				echo "Not applying non-fast forward update: $ref"
+			else
+				ok=1
+			fi
+		else
+			ok=1
+		fi
+	else
+		#forced, accept non-fast forward update
+		ok=1
+	fi
+	if [ ! -z "$ok" ] ; then
+		echo "updating: $ref to $sha1"
+		git-update-ref -m "git-unbundle update" $ref $sha1
+	fi
+done
-- 
1.5.0.34.g6afaa

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

> +# check that the prerequisites exist before unbundling.
> +for sha1 in $prereqs ; do
> +	git-rev-parse --verify $sha1 >& /dev/null || die "Prerequisite commit $sha1 not found."
> +done

If you are checking only boundary commits this is too weak a
check and unsafe (we used to have the same bug in http fetch
long time ago).  You have to make sure not only the object
exists, but also it is reachable by one of the refs.

One way to check that would be to ask "describe --all" if it can
find a ref that can reach that commit.  It would error out if
the commit object exists but is not reachable from any of the
refs.

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Linus Torvalds]

On Thu, 15 Feb 2007, Junio C Hamano wrote:
> 
> One way to check that would be to ask "describe --all" if it can
> find a ref that can reach that commit.  It would error out if
> the commit object exists but is not reachable from any of the
> refs.

Actually, more efficient and to the point:

	list-all-boundary-sha1s |
		git-rev-list --stdin --not --all

should return empty. Or something like that.

		Linus

[PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

Some workflows require coordinated development between repositories on
machines that can never be connected. This utility unpacks a bundle
containing objects and associated references (heads or tags) into the
current repository, effectively supporting git-push like operations
between disconnected systems.

Signed-off-by: Mark Levedahl <mdl123@verizon.net>
---
 git-unbundle.sh |   68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 68 insertions(+), 0 deletions(-)
 create mode 100755 git-unbundle.sh

diff --git a/git-unbundle.sh b/git-unbundle.sh
new file mode 100755
index 0000000..c947f15
--- /dev/null
+++ b/git-unbundle.sh
@@ -0,0 +1,68 @@
+#!/bin/sh
+# unpack a git-bundle file into current repository
+#
+# See git-bundle.
+
+USAGE="usage: git-unbundle [-f|--force] [file]"
+SUBDIRECTORY_OK=1
+. git-sh-setup
+
+bfile=bundle.tar
+force=
+while case "$#" in 0) break ;; esac
+do
+    case "$1" in
+    -f|--f|--fo|--for|--forc|--force)
+        force=1;;
+    -h|--h|--he|--hel|--help)
+        usage;;
+    -*)
+        die "unknown option: $1";;
+    *)
+        bfile="$1";;
+    esac
+    shift
+done
+
+test -e "$bfile" || die "cannot find $bfile"
+
+# Get prereqs and refs in one go to avoid a third tar invocation.
+# Have a header line with version, prereqs, then references.
+references=$(tar -xf "$bfile" --to-stdout references)
+refdata=${references##v1*prerequisites}
+test "$references" = "$refdata" && die "This doesn't look like a v1 bundle file."
+prereqs="${refdata%%references*}"
+refs="${refdata#*references}"
+
+# make sure prerequisites are available
+test -n "$prereqs" && printf "%s\n" $prereqs | \
+    git-rev-list --stdin --not --all || exit 1
+
+# get the pack file
+tar -xf "$bfile" --to-stdout pack | git-index-pack --stdin || exit 1
+
+# check each reference, avoid non-fast forward update unless forced
+printf "%s %s\n" $refs | while read sha1 ref ; do
+    ok=
+    if test -z "$force" ; then
+        # update only if non-fastforward
+        local=$(git-rev-parse --verify "$ref^0" 2>/dev/null)
+        if test -n "$local"  ; then
+            mb=$(git-merge-base $local $sha1)
+            if test "$mb" != "$local" ; then
+                echo "Not applying non-fast forward update: $ref"
+            else
+                ok=1
+            fi
+        else
+            ok=1
+        fi
+    else
+        #forced, accept non-fast forward update
+        ok=1
+    fi
+    if test -n "$ok" ; then
+        echo "updating: $ref to $sha1"
+        git-update-ref -m "git-unbundle update" $ref $sha1
+    fi
+done
-- 
1.5.0.rc4.375.gd0938-dirty

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Shawn O. Pearce]

Mark Levedahl <mdl123@verizon.net> wrote:
> +        # update only if non-fastforward
> +        local=$(git-rev-parse --verify "$ref^0" 2>/dev/null)
> +        if test -n "$local"  ; then
> +            mb=$(git-merge-base $local $sha1)
> +            if test "$mb" != "$local" ; then
> +                echo "Not applying non-fast forward update: $ref"
> +            else
> +                ok=1
> +            fi
> +        else
> +            ok=1
> +        fi
> +    else
> +        #forced, accept non-fast forward update
> +        ok=1
> +    fi
> +    if test -n "$ok" ; then
> +        echo "updating: $ref to $sha1"
> +        git-update-ref -m "git-unbundle update" $ref $sha1

What about passing $local as the final argument to update-ref,
so that the ref won't be modified if someone changed it while
an unbundle was running?  Sure its mostly a manual operation,
but imagine running it on a bare repository while someone else
is pushing into it...

-- 
Shawn.

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> Mark Levedahl <mdl123@verizon.net> wrote:
>> +        # update only if non-fastforward
>> +        local=$(git-rev-parse --verify "$ref^0" 2>/dev/null)
>> ...
>> +        git-update-ref -m "git-unbundle update" $ref $sha1
>
> What about passing $local as the final argument to update-ref,
> so that the ref won't be modified if someone changed it while
> an unbundle was running?  Sure its mostly a manual operation,
> but imagine running it on a bare repository while someone else
> is pushing into it...

The script already has $local at that point, so adding it to
update-ref is a no-cost change to make things safer.  I think it
makes sense.

But I have to wonder...  While someone else is _pushing_ into
it?  Why are _you_ sneakernetting, then?

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Shawn O. Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> "Shawn O. Pearce" <spearce@spearce.org> writes:
> > Mark Levedahl <mdl123@verizon.net> wrote:
> >> +        # update only if non-fastforward
> >> +        local=$(git-rev-parse --verify "$ref^0" 2>/dev/null)
> >> ...
> >> +        git-update-ref -m "git-unbundle update" $ref $sha1
> >
> > What about passing $local as the final argument to update-ref,
> > so that the ref won't be modified if someone changed it while
> > an unbundle was running?  Sure its mostly a manual operation,
> > but imagine running it on a bare repository while someone else
> > is pushing into it...
> 
> The script already has $local at that point, so adding it to
> update-ref is a no-cost change to make things safer.  I think it
> makes sense.

Actually that's only if --force was not given.  If --force was
given, rev-parse wasn't run, so local is not populated.  So maybe
its not trivial.
 
> But I have to wonder...  While someone else is _pushing_ into
> it?  Why are _you_ sneakernetting, then?

People do weird things.  I agree, its probably unlikely to ever
happen.  But give a user a length of rope, they will find a way to
hang themselves...

I can see someone trying to use an update hook with bundle/unbundle
to move stuff from one repository to another, despite the fact that
are better ways to do that.  Better that we fail when an update
might lose changes.

-- 
Shawn.

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Mark Levedahl]

Junio C Hamano wrote:
> Mark Levedahl <mdl123@verizon.net> writes:
>
>   
>> +# check that the prerequisites exist before unbundling.
>> +for sha1 in $prereqs ; do
>> +	git-rev-parse --verify $sha1 >& /dev/null || die "Prerequisite commit $sha1 not found."
>> +done
>>     
>
> If you are checking only boundary commits this is too weak a
> check and unsafe (we used to have the same bug in http fetch
> long time ago).  You have to make sure not only the object
> exists, but also it is reachable by one of the refs.
>
> One way to check that would be to ask "describe --all" if it can
> find a ref that can reach that commit.  It would error out if
> the commit object exists but is not reachable from any of the
> refs
I'm not sure I follow the logic here. Assuming the object is a 
prerequisite of the bundle and exists, it will be reachable from an 
updated reference once the bundle is applied, no? In any case, Linus' 
suggestion is very neat and fast, and I presume reliable.

Mark

Re: [PATCH] Add git-unbundle - unpack objects and references for disconnected transfer

[Junio C Hamano]

Mark Levedahl <mdl123@verizon.net> writes:

> I'm not sure I follow the logic here. Assuming the object is a
> prerequisite of the bundle and exists, it will be reachable from an
> updated reference once the bundle is applied, no?

No.  That's not the point.  Yes, the object might be reachable,
but you haven't proven that everything necessary to complete
that object is already in the object store.  We never trust
existence of an object alone when doing an operation that
updates a ref.  Having unreferenced objects in the object store
is a norm, but refs pointing at an object, some of whose
prerequisites are missing, is a crime.

For example, suppose an earlier http-fetch found that the
updated master should be at commit 'x', downloaded 'x', and you
killed the process with ^C.  It died without updating
refs/remotes/origin/master, which is good.  Then you have a
bundle to unpack that has 'x' as a boundary.

You would start with a repository like this:

  ---o---o---o...-...-...-...x
             ^origin.

where 'x' is a dangling commit, '-' are ones that you do not
have but you ought to have them in order for 'x' to be complete,
and 'o' are complete commits.  If the bundle boundary falls on
the 'x' commit, you cannot say "I have all that are needed to
complete this 'x'".  You only have 'x' but still lack its
prerequisites.  That's what it means never to trust the mere
existence of an object.

Linus's suggestion would make rev-list barf when 'x' cannot
reach any existing ref, which is a good solution for this
problem.

Also as Shawn pointed out, the script too heavily depends on GNU
tar.  Can we do something about it?  Requiring GNU tar is
probably fine if the script absolutely needs its extended
features, as long as people have ways to tweak the command name
in the script to point at their installed GNU tar (remember,
some systems have GNU tar under different name such as "gtar").

Although if we require GNU tar we might be better off placing
this under contrib/ not at the toplevel.  I dunno.