git-archive and submodules

git-archive and submodules

[André Caron]

Hi,

I've recently needed to create a source code archive as part of a
custom build target.  This repository uses submodules and I need to
include the submodlule's source code in the archives too.  However,
git-archive does not have any option to do so.

I've taken a quick look at the GMANE mailing list archives and it
seems this provoked quite a discussion in 2009 and that Lars Hjemli
even wrote a patch (in several flavors) for archive.c to include
submodule-aware processing.  The lastest source code at
`git.kernel.org` does not contain any traces of this patch (or
submodule aware logic for that matter).  The mailing list archives are
not very convenient to browse and I can't figure out what the status
on this submodule-aware git-archive idea is.  Has this idea been
completely rejected, or is it still work in progress?

In case you're wondering, I don't seem to be the only one needing to
do this.  I found at least two scripts on GitHub that implement this
logic with various levels of accuracy/completeness[1][2] and I've
hacked an icky Windows batch file version [3] for minimal Windows
support until this feature can be implemented in Git itself.  I don't
mind putting in some time into a real patch if I need to, but I'm
curious about the fact that this has been requested before, a patch
was proposed and yet this is still not merged 4 years later.

So, what's the status on a submodule-aware git-archive command?

Thanks,
André

[1]: https://github.com/meitar/git-archive-all.sh
[2]: https://github.com/Kentzo/git-archive-all
[3]: https://github.com/AndreLouisCaron/git-archive-all.bat

Re: git-archive and submodules

[Jens Lehmann]

Am 19.04.2012 22:10, schrieb André Caron:
> Hi,
> 
> I've recently needed to create a source code archive as part of a
> custom build target.  This repository uses submodules and I need to
> include the submodlule's source code in the archives too.  However,
> git-archive does not have any option to do so.
> 
> I've taken a quick look at the GMANE mailing list archives and it
> seems this provoked quite a discussion in 2009 and that Lars Hjemli
> even wrote a patch (in several flavors) for archive.c to include
> submodule-aware processing.  The lastest source code at
> `git.kernel.org` does not contain any traces of this patch (or
> submodule aware logic for that matter).  The mailing list archives are
> not very convenient to browse and I can't figure out what the status
> on this submodule-aware git-archive idea is.  Has this idea been
> completely rejected, or is it still work in progress?

The idea is not rejected, this would be a worthwhile addition. It has
been brought up again last May, but as far as I know it is stalled
since then:
http://comments.gmane.org/gmane.comp.version-control.git/172851

Fwd: git-archive and submodules

[André Caron]

Hi Robert,

I've recently raised the same submodule issue with git-archive you and
Lars have in the past.

What's the status on your application of the git-archive submodule
patch?  Have you applied the changes suggested by Lens Lehmann[1]?
Your Git fork on GitHub[2] seems to have only applied Lars' patch and
an unrelated SVN glob fix...

Since you've touched this only last year, I'd like to know where you
were at and I can see if I can pick up where you left off (unless you
want to finish yourself).

Thanks,
André

[1]: http://comments.gmane.org/gmane.comp.version-control.git/172851
[2]: https://github.com/darconeous/git


---------- Forwarded message ----------
From: Jens Lehmann <Jens.Lehmann@web.de>
Date: Fri, Apr 20, 2012 at 4:11 AM
Subject: Re: git-archive and submodules
To: André Caron <andre.l.caron@gmail.com>
Cc: git@vger.kernel.org


Am 19.04.2012 22:10, schrieb André Caron:
> Hi,
>
> I've recently needed to create a source code archive as part of a
> custom build target.  This repository uses submodules and I need to
> include the submodlule's source code in the archives too.  However,
> git-archive does not have any option to do so.
>
> I've taken a quick look at the GMANE mailing list archives and it
> seems this provoked quite a discussion in 2009 and that Lars Hjemli
> even wrote a patch (in several flavors) for archive.c to include
> submodule-aware processing.  The lastest source code at
> `git.kernel.org` does not contain any traces of this patch (or
> submodule aware logic for that matter).  The mailing list archives are
> not very convenient to browse and I can't figure out what the status
> on this submodule-aware git-archive idea is.  Has this idea been
> completely rejected, or is it still work in progress?

The idea is not rejected, this would be a worthwhile addition. It has
been brought up again last May, but as far as I know it is stalled
since then:
http://comments.gmane.org/gmane.comp.version-control.git/172851

Re: git-archive and submodules

[Robert Quattlebaum]

I got too busy to continue working to get it included. Please feel free to pick up where I left off. 

Sent from my iPad

On Apr 20, 2012, at 2:32 PM, André Caron <andre.l.caron@gmail.com> wrote:

> Hi Robert,
> 
> I've recently raised the same submodule issue with git-archive you and
> Lars have in the past.
> 
> What's the status on your application of the git-archive submodule
> patch?  Have you applied the changes suggested by Lens Lehmann[1]?
> Your Git fork on GitHub[2] seems to have only applied Lars' patch and
> an unrelated SVN glob fix...
> 
> Since you've touched this only last year, I'd like to know where you
> were at and I can see if I can pick up where you left off (unless you
> want to finish yourself).
> 
> Thanks,
> André
> 
> [1]: http://comments.gmane.org/gmane.comp.version-control.git/172851
> [2]: https://github.com/darconeous/git
> 
> 
> ---------- Forwarded message ----------
> From: Jens Lehmann <Jens.Lehmann@web.de>
> Date: Fri, Apr 20, 2012 at 4:11 AM
> Subject: Re: git-archive and submodules
> To: André Caron <andre.l.caron@gmail.com>
> Cc: git@vger.kernel.org
> 
> 
> Am 19.04.2012 22:10, schrieb André Caron:
>> Hi,
>> 
>> I've recently needed to create a source code archive as part of a
>> custom build target.  This repository uses submodules and I need to
>> include the submodlule's source code in the archives too.  However,
>> git-archive does not have any option to do so.
>> 
>> I've taken a quick look at the GMANE mailing list archives and it
>> seems this provoked quite a discussion in 2009 and that Lars Hjemli
>> even wrote a patch (in several flavors) for archive.c to include
>> submodule-aware processing.  The lastest source code at
>> `git.kernel.org` does not contain any traces of this patch (or
>> submodule aware logic for that matter).  The mailing list archives are
>> not very convenient to browse and I can't figure out what the status
>> on this submodule-aware git-archive idea is.  Has this idea been
>> completely rejected, or is it still work in progress?
> 
> The idea is not rejected, this would be a worthwhile addition. It has
> been brought up again last May, but as far as I know it is stalled
> since then:
> http://comments.gmane.org/gmane.comp.version-control.git/172851
> 

Re: git-archive and submodules

[Damien Regad]

Robert Quattlebaum <darco <at> deepdarc.com> writes:
> I got too busy to continue working to get it included. Please feel free to
pick up where I left off. 
> 
> On Apr 20, 2012, at 2:32 PM, André Caron <andre.l.caron <at> gmail.com> wrote:
> > Since you've touched this only last year, I'd like to know where you
> > were at and I can see if I can pick up where you left off (unless you
> > want to finish yourself).

Greetings

I was just wondering whether there been any progress on this topic since
last year... André ?

Cheers
Damien

Re: git-archive and submodules

[Amit Bakshi]

I wrote a simple bash script that does git-archive recursively with
submodules. It first does a full mirror clone of the repo and
submodules, so that subsequent calls are faster.

https://github.com/ambakshi/git-archiver


Amit


On Thu, Oct 10, 2013 at 7:09 AM, Damien Regad <dregad@mantisbt.org> wrote:
> Robert Quattlebaum <darco <at> deepdarc.com> writes:
>> I got too busy to continue working to get it included. Please feel free to
> pick up where I left off.
>>
>> On Apr 20, 2012, at 2:32 PM, André Caron <andre.l.caron <at> gmail.com> wrote:
>> > Since you've touched this only last year, I'd like to know where you
>> > were at and I can see if I can pick up where you left off (unless you
>> > want to finish yourself).
>
> Greetings
>
> I was just wondering whether there been any progress on this topic since
> last year... André ?
>
> Cheers
> Damien
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

git-archive and submodules

[Anatoly Borodin]

Hi All,


are there plans to add submodules support to git-archive?


-- 
Mit freundlichen Grüßen,
Anatoly Borodin

Re: git-archive and submodules

[Stefan Beller]

On Wed, Oct 26, 2016 at 1:37 PM, Anatoly Borodin
<anatoly.borodin@gmail.com> wrote:
> are there plans to add submodules support to git-archive?

plans by whom?

Git is a project with contributors from all over the place. (different
time zones,
people motivated by different means, i.e. we have the hobbiest that
scratches their
itch, we have paid people working on Git because their employer wants
them to work on Git,
there are other people (who like to) use Git in their work environment
and hack on it
in their spare time to make it awesome.)

AFAICT there are currently not a lot of people actively working on
submodule features,
though there is some history, e.g. Jens Lehmann maintains a wiki
specialised on submodules
https://github.com/jlehmann/git-submod-enhancements/wiki
and archive is mentioned there as one of the many "Issues still to be tackled".

Maybe you want to give it a try as you need it? I'd be happy to review any
submodule related code.

How to get started:

 * git clone https://github.com/git/git
 * Read (at least skim) Documentation/SubmittingPatches)
 * Look at builtin/archive.c as a starting point (cmd_archive is called
    when you call "git archive ...")
 * That leads to archive.c:write_archive, which calls parse_archive_args
  There we'd want to add an option there for recursing into submodules.
 * See write_archive_entry (still in archive.c) that mentions S_ISGITLINK
   Somewhere there you need to add code. :)

Thanks,
Stefan

git-archive and submodules

[Woody Gilk]

(Please note that I am not subscribed to the mailing list, so if you
want me to read your reply, it must be sent to
woody.gilk@kohanaphp.com)

I would find it hugely valuable if git-archive would support
submodules, rather than leaving empty files in the place of
submodules. Supporting submodules is critical to packaging releases
(combined with signed tags). With git-archive not support submodules,
it means that I have to manually package and sign each release, rather
than using git-tag.

This seems like it should be a fairly trivial thing to support,
possibly with the addition of a --with-submodules parameter for
git-archive to avoid backwards compatibility issues.

Thank you,
-Woody Gilk
Lead Developer of Kohana PHP Framework

Re: git-archive and submodules

[Avery Pennarun]

On Mon, Jul 20, 2009 at 12:22 PM, Woody Gilk<woody.gilk@kohanaphp.com> wrote:
> I would find it hugely valuable if git-archive would support
> submodules, rather than leaving empty files in the place of
> submodules. Supporting submodules is critical to packaging releases
> (combined with signed tags). With git-archive not support submodules,
> it means that I have to manually package and sign each release, rather
> than using git-tag.

<shameless-plug> This works already today if you use git-subtree[1]
instead of submodules. </shameless-plug>

Have fun,

Avery

[1] http://github.com/apenwarr/git-subtree

Re: git-archive and submodules

[Thomas Rast]

Woody Gilk wrote:
> I would find it hugely valuable if git-archive would support
> submodules, rather than leaving empty files in the place of
> submodules.

Such support was already in the pipeline before 1.6.2, see this
thread:

  http://thread.gmane.org/gmane.comp.version-control.git/107030

Lars' series is still available from the repo mentioned in one of the
messages

  git://hjemli.net/pub/git/git lh/traverse-gitlinks

but does not merge into master cleanly any more, because an earlier
version of the first commit (of now three) made it into 1.6.2.  For
your testing convenience, I rebased the other two commits on 'master'
and pushed them to

  git://repo.or.cz/git/trast.git lh/traverse-gitlinks-on-master

All conflicts were trivial, but I made a minor change: there's already
a t5001 from 8aece7f (archive test: attributes, 2009-04-18), so I
renamed it to t5002 in the tip commit.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

git-archive and submodules

[Whit Armstrong]

Is there a way to ask git-archive to archive the submodules of the
project as well?

I have a project that needs it's submoduels distributed with it.

Is anyone else using submodules in this way?

my project is: 	git://repo.or.cz/fts.git

It has a submodule: 	tslib at git://repo.or.cz/tslib.git

I create the archive by:
git archive --format=tar --prefix=fts/ HEAD | gzip > ~/fts.tar.gz

and when I go into the tslib submodule directory after expanding the
archive, it is empty.

-Whit

Re: git-archive and submodules

[Shawn O. Pearce]

Whit Armstrong <armstrong.whit@gmail.com> wrote:
> Is there a way to ask git-archive to archive the submodules of the
> project as well?
> 
> I have a project that needs it's submoduels distributed with it.

No.

Patches welcome.  :-)
 
We've talked about supporting it, and wouldn't mind having the
tool do it for exactly the reason you mention, but thus far a
patch has not been written to implement that.

-- 
Shawn.

Re: git-archive and submodules

[Whit Armstrong]

as a first patch that might be a bit ambitious for me, but I will give it a try.

anyway, it's good to know that there is additional support for this behavior.

-Whit


On Sat, Mar 29, 2008 at 9:52 AM, Shawn O. Pearce <spearce@spearce.org> wrote:
> Whit Armstrong <armstrong.whit@gmail.com> wrote:
>  > Is there a way to ask git-archive to archive the submodules of the
>  > project as well?
>  >
>  > I have a project that needs it's submoduels distributed with it.
>
>  No.
>
>  Patches welcome.  :-)
>
>  We've talked about supporting it, and wouldn't mind having the
>  tool do it for exactly the reason you mention, but thus far a
>  patch has not been written to implement that.
>
>  --
>  Shawn.
>

Re: git-archive and submodules

[CJ van den Berg]

[-- Attachment #1.1: Type: text/plain, Size: 959 bytes --]

On Sat, Mar 29, 2008 at 09:52:02AM -0400, Shawn O. Pearce wrote:
> Whit Armstrong <armstrong.whit@gmail.com> wrote:
> > Is there a way to ask git-archive to archive the submodules of the
> > project as well?
> > 
> > I have a project that needs it's submoduels distributed with it.
> 
> No.
> 
> Patches welcome.  :-)
>  
> We've talked about supporting it, and wouldn't mind having the
> tool do it for exactly the reason you mention, but thus far a
> patch has not been written to implement that.

Here's a script I use at work that calls git-archive recursively for
submodules and builds a single tarball for everything. If there is
interest in having something like this in git proper I could put some
work into cleaning it up for general use. Perhaps something like this
should be integrated into git-archive directly. Comments and suggestions
are welcome.

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com

[-- Attachment #1.2: git-archive-recursive --]
[-- Type: text/plain, Size: 3184 bytes --]

#!/bin/bash

set -e

TMP_DIR=$(mktemp -t -d $(basename $0).$USER.XXXXX)
GITMODULES_FILE=${TMP_DIR}/gitmodules
THIS_TAR=${TMP_DIR}/this.tar
SUBMODULE_TAR=${TMP_DIR}/submodule.tar

function cleanup() {
        rm -rf $TMP_DIR
}

function die() {
        cleanup
                exit 1
}

trap die SIGINT
trap die SIGHUP
trap die SIGTERM
trap die SIGQUIT
trap die ERR
trap cleanup EXIT

function usage() {
    echo "Build an archive of a repository and all submodules." >&2
    echo "" >&2
    echo "usage: $(basename $0) [-r n] repo treeish [prefix]" >&2
    echo "  repo      Repository path or URL." >&2
    echo "  treeish   Version to archive." >&2
    echo "  -r n      Maximum recursion level. Default is 1." >&2
    echo "  -h        Show this message." >&2
    echo "" >&2
    exit 1
}

while [ "${1:0:1}" == "-" ]
do
    if [ "$1" == "-h" ] ; then
        usage
    fi
    if [ "$1" == "-r" ] ; then
        LEVEL=$2
        shift
        if [ -z "$LEVEL" ] ; then
            usage
        fi
        if [ ! $LEVEL -lt 0 ] ; then
            echo -n "" >&2
        else
            usage
        fi
    fi
    shift
done

if [ $# -lt 2 -o $# -gt 3 ] ; then
    usage
fi

export GIT_DIR=$1
REV="$2"
PREFIX="$3"
if [ -z "$LEVEL" ] ; then
    LEVEL=1
fi

if echo $GIT_DIR | grep -q ":"
then
    REMOTE_HOST=$(echo $GIT_DIR | cut -d ":" -f 1)
    REMOTE_PATH=$(echo $GIT_DIR | cut -d ":" -f 2)
    if [ "$REMOTE_HOST" == "$(hostname)" ] ; then
        export GIT_DIR=${REMOTE_PATH}
    else
        exec ssh -n $REMOTE_HOST $(basename $0) -r $LEVEL $REMOTE_PATH $REV $PREFIX
    fi
fi

if [ "${GIT_DIR:0:2}" == "~/" ] ; then
    export GIT_DIR=/home/${USER}/${GIT_DIR#\~/}
elif [ "${GIT_DIR:0:1}" == "~" ] ; then
    export GIT_DIR=/home/${GIT_DIR#\~}
fi

REV_NAME=$(git describe ${REV} 2> /dev/null || true)
if [ -z "$REV_NAME" ] ; then
    REV_NAME=$(git rev-parse ${REV})
fi

if [ -z "$PREFIX" ] ; then
    PREFIX=$REV_NAME
fi

echo >&2 ${PREFIX} from $(git-make-repo-url) \(${REV_NAME}\)
git archive --prefix=${PREFIX}/ $REV > $THIS_TAR

mkdir -p $TMP_DIR/$PREFIX
echo ${REV_NAME}> $TMP_DIR/$PREFIX/GITVERSION
echo $(git-make-repo-url)> $TMP_DIR/$PREFIX/GITSOURCE
(
    cd $TMP_DIR
    tar -rf $THIS_TAR $PREFIX
)

if [ $LEVEL -gt 0 ]
then
    if git show $REV:.gitmodules > $GITMODULES_FILE 2> /dev/null
    then
        LEVEL=$(($LEVEL - 1))
        git ls-tree -r "$REV" | grep '^160000 ' |
        while read mode type sha1 path
        do
            (
            submodule_name=$(git config --file $GITMODULES_FILE --list | \
                              grep "^submodule\..*\.path=$path" | cut -d . -f 2)
            if [ -z "$submodule_name" ] ; then
                echo >&2 -n "fatal: No submodule mapping found in .gitmodules "
                echo >&2    "for path '$path'"
                exit 1
            fi
            url=$(git config --file $GITMODULES_FILE \
                            submodule."$submodule_name".url || exit 1)
            $(basename $0) -r $LEVEL $url $sha1 $PREFIX/$path > $SUBMODULE_TAR || exit 1
            tar --concatenate --file=$THIS_TAR $SUBMODULE_TAR || exit 1
            ) < /dev/null
        done
    fi
fi
cat $THIS_TAR

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]