[Qemu-devel] [PATCH v2] scripts: use git archive in archive-source

[Qemu-devel] [PATCH v2] scripts: use git archive in archive-source

[Gerd Hoffmann]

Use git archive to create tarballs of qemu and submodules instead of
cloning the repository and the submodules.  This is a order of magnitude
faster because it doesn't fetch the submodules from the internet each
time the script runs.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---

Notes:
    v2: don't run git submodule update (except for uninitialized submodules)

 scripts/archive-source.sh | 58 ++++++++++++++++++-----------------------------
 1 file changed, 22 insertions(+), 36 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 6eed2a29bd..b9711059f9 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -19,8 +19,7 @@ if test $# -lt 1; then
 fi
 
 tar_file=$(realpath "$1")
-list_file="${tar_file}.list"
-vroot_dir="${tar_file}.vroot"
+sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar")
 
 # We want a predictable list of submodules for builds, that is
 # independent of what the developer currently has initialized
@@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot"
 # different to the host OS.
 submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
 
-trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
+trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15
 
 if git diff-index --quiet HEAD -- &>/dev/null
 then
@@ -36,38 +35,25 @@ then
 else
     HEAD=$(git stash create)
 fi
-git clone --shared . "$vroot_dir"
-test $? -ne 0 && error "failed to clone into '$vroot_dir'"
-
-cd "$vroot_dir"
-test $? -ne 0 && error "failed to change into '$vroot_dir'"
-
-git checkout $HEAD
-test $? -ne 0 && error "failed to checkout $HEAD revision"
-
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
 for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
+	status="$(git submodule status "$sm")"
+	smhash="${status# }"
+	smhash="${smhash#+}"
+	smhash="${smhash#-}"
+	smhash="${smhash%% *}"
+	case "$status" in
+	    -*)
+		echo "NOTICE: fetching submodule $sm"
+		git submodule update --init "$sm"
+		;;
+	    +*)
+		echo "WARNING: submodule $sm is out of sync"
+		;;
+	esac
+	(cd $sm; git archive --format tar --prefix "$sm/" $smhash) > "$sub_file"
+	test $? -ne 0 && error "failed to archive submodule $sm ($smhash)"
+	tar --concatenate --file "$tar_file" "$sub_file"
+	test $? -ne 0 && error "failed append submodule $sm to $tar_file"
 done
-
-if test -n "$submodules"; then
-    {
-        git ls-files || error "git ls-files failed"
-        for sm in $submodules; do
-            (cd $sm; git ls-files) | sed "s:^:$sm/:"
-            if test "${PIPESTATUS[*]}" != "0 0"; then
-                error "git ls-files in submodule $sm failed"
-            fi
-        done
-    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
-else
-    git ls-files > "$list_file"
-fi
-
-if test $? -ne 0; then
-    error "failed to generate list file"
-fi
-
-tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
-
-exit 0
-- 
2.9.3

Re: [Qemu-devel] [PATCH v2] scripts: use git archive in archive-source

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190131075132.17706-1-kraxel@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH v2] scripts: use git archive in archive-source
Message-id: 20190131075132.17706-1-kraxel@redhat.com
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Submodule 'capstone' (https://git.qemu.org/git/capstone.git) registered for path 'capstone'
Submodule 'dtc' (https://git.qemu.org/git/dtc.git) registered for path 'dtc'
Submodule 'roms/QemuMacDrivers' (https://git.qemu.org/git/QemuMacDrivers.git) registered for path 'roms/QemuMacDrivers'
Submodule 'roms/SLOF' (https://git.qemu.org/git/SLOF.git) registered for path 'roms/SLOF'
Submodule 'roms/ipxe' (https://git.qemu.org/git/ipxe.git) registered for path 'roms/ipxe'
Submodule 'roms/openbios' (https://git.qemu.org/git/openbios.git) registered for path 'roms/openbios'
Submodule 'roms/openhackware' (https://git.qemu.org/git/openhackware.git) registered for path 'roms/openhackware'
Submodule 'roms/qemu-palcode' (https://git.qemu.org/git/qemu-palcode.git) registered for path 'roms/qemu-palcode'
Submodule 'roms/seabios' (https://git.qemu.org/git/seabios.git/) registered for path 'roms/seabios'
Submodule 'roms/seabios-hppa' (https://github.com/hdeller/seabios-hppa.git) registered for path 'roms/seabios-hppa'
Submodule 'roms/sgabios' (https://git.qemu.org/git/sgabios.git) registered for path 'roms/sgabios'
Submodule 'roms/skiboot' (https://git.qemu.org/git/skiboot.git) registered for path 'roms/skiboot'
Submodule 'roms/u-boot' (https://git.qemu.org/git/u-boot.git) registered for path 'roms/u-boot'
Submodule 'roms/u-boot-sam460ex' (https://git.qemu.org/git/u-boot-sam460ex.git) registered for path 'roms/u-boot-sam460ex'
Submodule 'tests/fp/berkeley-softfloat-3' (https://github.com/cota/berkeley-softfloat-3) registered for path 'tests/fp/berkeley-softfloat-3'
Submodule 'tests/fp/berkeley-testfloat-3' (https://github.com/cota/berkeley-testfloat-3) registered for path 'tests/fp/berkeley-testfloat-3'
Submodule 'ui/keycodemapdb' (https://git.qemu.org/git/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into 'capstone'...
Submodule path 'capstone': checked out '22ead3e0bfdb87516656453336160e0a37b066bf'
Cloning into 'dtc'...
Submodule path 'dtc': checked out '88f18909db731a627456f26d779445f84e449536'
Cloning into 'roms/QemuMacDrivers'...
Submodule path 'roms/QemuMacDrivers': checked out 'd4e7d7ac663fcb55f1b93575445fcbca372f17a7'
Cloning into 'roms/SLOF'...
Submodule path 'roms/SLOF': checked out '9b7ab2fa020341dee8bf9df6c9cf40003e0136df'
Cloning into 'roms/ipxe'...
Submodule path 'roms/ipxe': checked out 'de4565cbe76ea9f7913a01f331be3ee901bb6e17'
Cloning into 'roms/openbios'...
Submodule path 'roms/openbios': checked out '441a84d3a642a10b948369c63f32367e8ff6395b'
Cloning into 'roms/openhackware'...
Submodule path 'roms/openhackware': checked out 'c559da7c8eec5e45ef1f67978827af6f0b9546f5'
Cloning into 'roms/qemu-palcode'...
Submodule path 'roms/qemu-palcode': checked out '51c237d7e20d05100eacadee2f61abc17e6bc097'
Cloning into 'roms/seabios'...
Submodule path 'roms/seabios': checked out 'a698c8995ffb2838296ec284fe3c4ad33dfca307'
Cloning into 'roms/seabios-hppa'...
Submodule path 'roms/seabios-hppa': checked out '1ef99a01572c2581c30e16e6fe69e9ea2ef92ce0'
Cloning into 'roms/sgabios'...
Submodule path 'roms/sgabios': checked out 'cbaee52287e5f32373181cff50a00b6c4ac9015a'
Cloning into 'roms/skiboot'...
Submodule path 'roms/skiboot': checked out 'e0ee24c27a172bcf482f6f2bc905e6211c134bcc'
Cloning into 'roms/u-boot'...
Submodule path 'roms/u-boot': checked out 'd85ca029f257b53a96da6c2fb421e78a003a9943'
Cloning into 'roms/u-boot-sam460ex'...
Submodule path 'roms/u-boot-sam460ex': checked out '60b3916f33e617a815973c5a6df77055b2e3a588'
Cloning into 'tests/fp/berkeley-softfloat-3'...
Submodule path 'tests/fp/berkeley-softfloat-3': checked out 'b64af41c3276f97f0e181920400ee056b9c88037'
Cloning into 'tests/fp/berkeley-testfloat-3'...
Submodule path 'tests/fp/berkeley-testfloat-3': checked out '5a59dcec19327396a011a17fd924aed4fec416b3'
Cloning into 'ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '6b3d716e2b6472eb7189d3220552280ef3d832ce'
Switched to a new branch 'test'
abfc980 scripts: use git archive in archive-source

=== OUTPUT BEGIN ===
ERROR: code indent should never use tabs
#56: FILE: scripts/archive-source.sh:41:
+^Istatus="$(git submodule status "$sm")"$

ERROR: code indent should never use tabs
#57: FILE: scripts/archive-source.sh:42:
+^Ismhash="${status# }"$

ERROR: code indent should never use tabs
#58: FILE: scripts/archive-source.sh:43:
+^Ismhash="${smhash#+}"$

ERROR: code indent should never use tabs
#59: FILE: scripts/archive-source.sh:44:
+^Ismhash="${smhash#-}"$

ERROR: code indent should never use tabs
#60: FILE: scripts/archive-source.sh:45:
+^Ismhash="${smhash%% *}"$

ERROR: code indent should never use tabs
#61: FILE: scripts/archive-source.sh:46:
+^Icase "$status" in$

ERROR: code indent should never use tabs
#62: FILE: scripts/archive-source.sh:47:
+^I    -*)$

ERROR: code indent should never use tabs
#63: FILE: scripts/archive-source.sh:48:
+^I^Iecho "NOTICE: fetching submodule $sm"$

ERROR: code indent should never use tabs
#64: FILE: scripts/archive-source.sh:49:
+^I^Igit submodule update --init "$sm"$

ERROR: code indent should never use tabs
#65: FILE: scripts/archive-source.sh:50:
+^I^I;;$

ERROR: code indent should never use tabs
#66: FILE: scripts/archive-source.sh:51:
+^I    +*)$

ERROR: code indent should never use tabs
#67: FILE: scripts/archive-source.sh:52:
+^I^Iecho "WARNING: submodule $sm is out of sync"$

ERROR: code indent should never use tabs
#68: FILE: scripts/archive-source.sh:53:
+^I^I;;$

ERROR: code indent should never use tabs
#69: FILE: scripts/archive-source.sh:54:
+^Iesac$

ERROR: code indent should never use tabs
#70: FILE: scripts/archive-source.sh:55:
+^I(cd $sm; git archive --format tar --prefix "$sm/" $smhash) > "$sub_file"$

ERROR: code indent should never use tabs
#71: FILE: scripts/archive-source.sh:56:
+^Itest $? -ne 0 && error "failed to archive submodule $sm ($smhash)"$

ERROR: code indent should never use tabs
#72: FILE: scripts/archive-source.sh:57:
+^Itar --concatenate --file "$tar_file" "$sub_file"$

ERROR: code indent should never use tabs
#73: FILE: scripts/archive-source.sh:58:
+^Itest $? -ne 0 && error "failed append submodule $sm to $tar_file"$

total: 18 errors, 0 warnings, 53 lines checked

Commit abfc980abb39 (scripts: use git archive in archive-source) has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190131075132.17706-1-kraxel@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com