Running git-archive recursively over submodules

Running git-archive recursively over submodules

[Thomas Rast]

I'm posting this for posterity.  It came out of an IRC discussion last
night.  Google "git archive submodules" currently finds two scripts
for the same task:

* http://kerneltrap.org/mailarchive/git/2008/3/29/1294614
  Can pack arbitrary revisions, but seems to be highly specific to the
  poster's setup.

* http://wiki.github.com/meitar/git-archive-all.sh/
  Appears to work with normal repos (I haven't really tested) but only
  packs HEAD.

Both have in common that they run git-archive in each submodule, and
then merge the resulting tars.

The script below follows a different approach, based on the idea that
all the objects are already there, you just need to help git find
them.  This of course assumes that you have the submodules checked out
(or at least cloned).  It builds an index that contains a flattened
version of the repository and all submodules.  This works for any
revision you specify.  The downside is that this does not respect
.gitattributes.

The same trick can be used for other commands, so if people find this
useful it could be added to git-submodule.

Warning: Error detection and such are left as an exercise to the user.

-- 8< --
#!/bin/sh

export revision="$1"

export GIT_INDEX_FILE=".git/tmpindex"
rm -f "$GIT_INDEX_FILE"

git read-tree $revision

export up="$(pwd)"

read_one_level () {
	export GIT_ALTERNATE_OBJECT_DIRECTORIES="$GIT_ALTERNATE_OBJECT_DIRECTORIES":$(
	    git submodule foreach 'echo "$up/$path/.git/objects"' |
	    grep -E -v '^(Entering|No submodule mapping found)' |
	    tr '\n' : |
	    sed 's/:$//'
	)

	git submodule foreach '
		cd "$up"
		subcommit=$(git rev-parse :"$path")
		git rm --cached "$path"
		git read-tree -i --prefix="$path/" $subcommit
	' >/dev/null
}

while git ls-files -s | grep -q ^160000; do
    read_one_level
done

git archive --format=tar $(git write-tree)

rm -f "$GIT_INDEX_FILE"
-- >8 --

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Running git-archive recursively over submodules

[Michael J Gruber]

Thomas Rast venit, vidit, dixit 15.02.2010 21:56:
> I'm posting this for posterity.  It came out of an IRC discussion last
> night.  Google "git archive submodules" currently finds two scripts
> for the same task:
> 
> * http://kerneltrap.org/mailarchive/git/2008/3/29/1294614
>   Can pack arbitrary revisions, but seems to be highly specific to the
>   poster's setup.
> 
> * http://wiki.github.com/meitar/git-archive-all.sh/
>   Appears to work with normal repos (I haven't really tested) but only
>   packs HEAD.
> 
> Both have in common that they run git-archive in each submodule, and
> then merge the resulting tars.
> 
> The script below follows a different approach, based on the idea that
> all the objects are already there, you just need to help git find
> them.  This of course assumes that you have the submodules checked out
> (or at least cloned).  It builds an index that contains a flattened
> version of the repository and all submodules.  This works for any
> revision you specify.  The downside is that this does not respect
> .gitattributes.
> 
> The same trick can be used for other commands, so if people find this
> useful it could be added to git-submodule.
> 
> Warning: Error detection and such are left as an exercise to the user.
> 
> -- 8< --
> #!/bin/sh
> 
> export revision="$1"
> 
> export GIT_INDEX_FILE=".git/tmpindex"
> rm -f "$GIT_INDEX_FILE"
> 
> git read-tree $revision
> 
> export up="$(pwd)"
> 
> read_one_level () {
> 	export GIT_ALTERNATE_OBJECT_DIRECTORIES="$GIT_ALTERNATE_OBJECT_DIRECTORIES":$(
> 	    git submodule foreach 'echo "$up/$path/.git/objects"' |
> 	    grep -E -v '^(Entering|No submodule mapping found)' |
> 	    tr '\n' : |
> 	    sed 's/:$//'
> 	)
> 
> 	git submodule foreach '
> 		cd "$up"
> 		subcommit=$(git rev-parse :"$path")
> 		git rm --cached "$path"
> 		git read-tree -i --prefix="$path/" $subcommit
> 	' >/dev/null
> }
> 
> while git ls-files -s | grep -q ^160000; do
>     read_one_level
> done

Am I blind, not seeing a recursion, or is there simply none? I reckon
you're reading the super project and one level of submodules into the
temporary index - which is fine for most purposes, of course. I would
say this is iterating over the submodules of the super project.

> git archive --format=tar $(git write-tree)
> 
> rm -f "$GIT_INDEX_FILE"
> -- >8 --
> 

Re: Running git-archive recursively over submodules

[Thomas Rast]

On Tuesday 16 February 2010 08:38:08 Michael J Gruber wrote:
> Thomas Rast venit, vidit, dixit 15.02.2010 21:56:
> > while git ls-files -s | grep -q ^160000; do
> >     read_one_level
> > done
> 
> Am I blind, not seeing a recursion, or is there simply none? I reckon
> you're reading the super project and one level of submodules into the
> temporary index - which is fine for most purposes, of course. I would
> say this is iterating over the submodules of the super project.

Well, the while condition reads "while there are submodule entries in
my index".  The read_one_level function then replaces those submodules
with their corresponding trees, which of course may again contain
submodule entries.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Running git-archive recursively over submodules

[Michael J Gruber]

Thomas Rast venit, vidit, dixit 16.02.2010 11:01:
> On Tuesday 16 February 2010 08:38:08 Michael J Gruber wrote:
>> Thomas Rast venit, vidit, dixit 15.02.2010 21:56:
>>> while git ls-files -s | grep -q ^160000; do
>>>     read_one_level
>>> done
>>
>> Am I blind, not seeing a recursion, or is there simply none? I reckon
>> you're reading the super project and one level of submodules into the
>> temporary index - which is fine for most purposes, of course. I would
>> say this is iterating over the submodules of the super project.
> 
> Well, the while condition reads "while there are submodule entries in
> my index".  The read_one_level function then replaces those submodules
> with their corresponding trees, which of course may again contain
> submodule entries.
> 

Gotcha, and the rm --cached in foreach makes sure that submodule entries
disappear from the index once their contents have been read into it, so
that the recursion stops when there's no deeper level. Clever.

Thanks for helping my for/while blindness (I guess I was thinking "for").

Cheers,
Michael