Re: [PATCH] allow git-bundle to create bottomless bundle

["Mark Levedahl" <mlevedahl@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

On 8/9/07, Junio C Hamano <gitster@pobox.com> wrote:
> Mark Levedahl <mlevedahl@gmail.com> writes:
>
> > Junio C Hamano wrote:
>
> Actually, there is another bug nearby.
>
Curious... I did a long time ago get git-bundle.sh to write out the
format Dscho finally implemented working in Linux and Cygwin. With
your latest patch, both produce identical bundles for a number of
cases, but the timing is very curious:

*builtin*
git>time git-bundle create foo refs/tags/v1.0.3
Generating pack...
Done counting 12288 objects.
Deltifying 12288 objects...
 100% (12288/12288) done
Writing 12288 objects...
 100% (12288/12288) done
Total 12288 (delta 8435), reused 11542 (delta 7961)

real    0m41.953s
user    0m5.060s
sys     0m39.685s

*shell script*
git>time git-bundle2 create foo2 refs/tags/v1.0.3
Generating pack...
Done counting 12288 objects.
Deltifying 12288 objects...
 100% (12288/12288) done
Total 12288 (delta 8435), reused 11542 (delta 7961)
Created foo2

real    0m2.453s
user    0m1.842s
sys     0m1.190s
git>diff foo foo2
git>

Since when is shell 20 times faster than a builtin? Ok, those results
are on Cygwin. On Linux, I get  1.21 sec for the builtin vs 0.933 for
the shell script. Not as dramatic, but the script is still faster. So,
I think this qualifies as a bug. I'm attaching the script (for info
only, I'm not suggesting to replace the builtin), as I can't usefully
put it inline from gmail.

The timing difference seem to be in the rev walking (by eyeball,
packfile generation time is the same), so something the shell script
is doing to get the rev list is obviously more efficient than how the
builtin works for this case.

Mark

[-- Attachment #2: git-bundle2 --]
[-- Type: application/octet-stream, Size: 4705 bytes --]

#!/bin/sh
# Basic handler for bundle files to connect repositories via sneakernet.
# Invocation must include action.
# This function can create a bundle or provide information on an existing bundle
# supporting git-fetch, git-pull, and git-ls-remote

# bundle format is:
# line1 - version
# -sha1 commit msg  : prerequisites
# sha1 name         ; refs
# <blank line>
# pack file

USAGE='git-bundle ( create <bundle> [<rev-list-args>...] |
         verify | list-heads | unbundle <bundle> )'

SUBDIRECTORY_OK=1
. git-sh-setup
LF='
'

verify() {
    # Check bundle version
    test "$version" = "# v2 git bundle" ||
        die "$bfile is not a v2 git bundle"

    # do fast check, then if any prereqs are missing then go line by line
    # to be verbose about the errors
    test -z "$prereqs" && return 0
    if bad=$(echo "$prereqs" | cut -b-40 | git-rev-list --stdin --not --all 2>&1) &&
    test -z "$bad" ; then
        return 0
    else
        echo >/dev/stderr "error: $bfile requires the following missing commits:"
        echo "$prereqs" |
        while read sha1 comment ; do
            if ! missing=$(git-rev-list $sha1 --not --all 2>&1) || test -n "$missing" ; then
                echo >/dev/stderr "$sha1 $comment"
            fi
        done
        return 1
    fi
}

# list all refs or just a subset
list_heads() {
    if test -z "$*" ; then
        echo "$refs"
    else
        echo "$refs" |
        while read sha1 ref ; do
            for arg in $* ; do
                if test "${ref%$arg}" != "$ref" ; then
                    echo "$sha1 $ref"
                    break
                fi
            done
        done
    fi
}

# create a bundle
create() {
    unknown=$(git-rev-parse --no-revs $args)
    test -z "$unknown" || die "unknown option: $unknown"
    gitrevargs=$(git-rev-parse --symbolic --revs-only $args) || exit 1

    # find the refs to carry along and get sha1s for each.
    refs=
    boundaryargs=
    basis=
    for arg in $gitrevargs ; do
        #ignore options and basis refs, get full ref name for things
        # we will transport rejecting anything ambiguous (e.g., user
        # gives master, have heads/master and remotes/origin/master, we
        # keep the former).
        case "$arg" in
            ^*)
                boundaryargs="$boundaryargs $arg"
                basis="$basis ${arg/^/}";;
            -*)
                boundaryargs="$boundaryargs $arg";;
            *)
                refs="$refs$LF$(git show-ref $arg | head -n1)";;
        esac
    done
    test -z "$refs" && die "No references specified, I don't know what to bundle."
	sha1refs=$(echo "$refs" | cut -b-40)

    # get prerequisite commits this bundle depends upon
    prereqs=$(git-rev-list --boundary $boundaryargs $sha1refs | sed -ne 's/^-//p')
    if test -n "$basis" ; then
        prereqs=$(printf "%s\n" $prereqs $(git-rev-parse $basis) | sort | uniq)
    fi
	
    # create refs and pack. We do this in a subshell to avoid using >> to append,
    # as that induces crlf breakage under Cygwin.
    (
        echo "# v2 git bundle" &&
        (for p in $prereqs ; do
            printf "%s\n" "-$(git-rev-list --pretty=one --max-count=1 $p)"
        done) &&
        git-show-ref $refs &&
        echo "" &&
        (printf "%s\n" $sha1refs &&
        test -n "$prereqs" && printf "^%s\n" $prereqs ) |
        git pack-objects --thin --stdout --progress
    ) > "$bfile"
    if test $? -gt 0; then
        rm -fv "$bfile"
    else
        echo "Created $bfile"
    fi
}

# parse inputs, decide what to do
args=
action=
while test -n "$1" ; do
    case $1 in
        create|list-heads|unbundle|verify|show)
            action=$1
            shift
            bfile=$1
            test -z "$bfile" && die "$action requires filename";;
        *)
            args="$args $1";;
    esac
    shift
done
test -z "$action" && die "No action given, what should I do?"

if test "$action" = "create" ; then
    create
else
    test -f "$bfile" || die "Could not find $bfile"

    # get the header once, verify all is ok
    header=$(sed -e '/^$/,$d' "$bfile")
    refs=$(echo "$header" | sed -e '1d;/^-/d')
    prereqs=$(echo "$header" | sed -ne 's/^-//p')
    version=$(echo "$header" | head -n1)

    echo $version

    case $action in
        verify)
            verify && echo "$bfile is ok";;

        list-heads)
            list_heads $args;;

        show)
            echo "$header" | sed -ne "/^-/p"
            echo "$refs";;

        unbundle)
            verify || exit 1
            sed -be '1,/^$/d' "$bfile" | git-index-pack --stdin --fix-thin ||
                die "error: $bfile has a corrupted pack file"
            list_heads $args;;
    esac
fi