[Qemu-devel] [PATCH] scripts: use git archive in archive-source

[Qemu-devel] [PATCH] scripts: use git archive in archive-source

[Gerd Hoffmann]

Use git archive to create tarballs of qemu and submodules instead of
cloning the repository and the submodules.  This is a order of magnitude
faster because it doesn't fetch the submodules from the internet each
time the script runs.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 scripts/archive-source.sh | 46 ++++++++++------------------------------------
 1 file changed, 10 insertions(+), 36 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 6eed2a29bd..beae968f80 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -19,8 +19,7 @@ if test $# -lt 1; then
 fi
 
 tar_file=$(realpath "$1")
-list_file="${tar_file}.list"
-vroot_dir="${tar_file}.vroot"
+sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar")
 
 # We want a predictable list of submodules for builds, that is
 # independent of what the developer currently has initialized
@@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot"
 # different to the host OS.
 submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
 
-trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
+trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15
 
 if git diff-index --quiet HEAD -- &>/dev/null
 then
@@ -36,38 +35,13 @@ then
 else
     HEAD=$(git stash create)
 fi
-git clone --shared . "$vroot_dir"
-test $? -ne 0 && error "failed to clone into '$vroot_dir'"
-
-cd "$vroot_dir"
-test $? -ne 0 && error "failed to change into '$vroot_dir'"
-
-git checkout $HEAD
-test $? -ne 0 && error "failed to checkout $HEAD revision"
-
+git archive --format tar $HEAD > "$tar_file"
+test $? -ne 0 && error "failed to archive qemu"
 for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
+	git submodule update --init "$sm"
+	test $? -ne 0 && error "failed to update submodule $sm"
+	(cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"
+	test $? -ne 0 && error "failed to archive submodule $sm"
+	tar --concatenate --file "$tar_file" "$sub_file"
+	test $? -ne 0 && error "failed append submodule $sm to $tar_file"
 done
-
-if test -n "$submodules"; then
-    {
-        git ls-files || error "git ls-files failed"
-        for sm in $submodules; do
-            (cd $sm; git ls-files) | sed "s:^:$sm/:"
-            if test "${PIPESTATUS[*]}" != "0 0"; then
-                error "git ls-files in submodule $sm failed"
-            fi
-        done
-    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
-else
-    git ls-files > "$list_file"
-fi
-
-if test $? -ne 0; then
-    error "failed to generate list file"
-fi
-
-tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
-
-exit 0
-- 
2.9.3

Re: [Qemu-devel] [PATCH] scripts: use git archive in archive-source

[Daniel P. Berrangé]

On Wed, Jan 30, 2019 at 07:55:30AM +0100, Gerd Hoffmann wrote:
> Use git archive to create tarballs of qemu and submodules instead of
> cloning the repository and the submodules.  This is a order of magnitude
> faster because it doesn't fetch the submodules from the internet each
> time the script runs.

There's another patch addressing the same problem here, which takes
the approach of 'git clone' on the submodules to avoid pulling them
from the internet each time:

  https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06588.html

> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  scripts/archive-source.sh | 46 ++++++++++------------------------------------
>  1 file changed, 10 insertions(+), 36 deletions(-)
> 
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index 6eed2a29bd..beae968f80 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -19,8 +19,7 @@ if test $# -lt 1; then
>  fi
>  
>  tar_file=$(realpath "$1")
> -list_file="${tar_file}.list"
> -vroot_dir="${tar_file}.vroot"
> +sub_file=$(mktemp "${tar_file%.tar}.sub.XXXXXXXX.tar")
>  
>  # We want a predictable list of submodules for builds, that is
>  # independent of what the developer currently has initialized
> @@ -28,7 +27,7 @@ vroot_dir="${tar_file}.vroot"
>  # different to the host OS.
>  submodules="dtc ui/keycodemapdb tests/fp/berkeley-softfloat-3 tests/fp/berkeley-testfloat-3"
>  
> -trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
> +trap "status=$?; rm -rf \"$sub_file\" ; exit \$status" 0 1 2 3 15
>  
>  if git diff-index --quiet HEAD -- &>/dev/null
>  then
> @@ -36,38 +35,13 @@ then
>  else
>      HEAD=$(git stash create)
>  fi
> -git clone --shared . "$vroot_dir"
> -test $? -ne 0 && error "failed to clone into '$vroot_dir'"
> -
> -cd "$vroot_dir"
> -test $? -ne 0 && error "failed to change into '$vroot_dir'"
> -
> -git checkout $HEAD
> -test $? -ne 0 && error "failed to checkout $HEAD revision"
> -
> +git archive --format tar $HEAD > "$tar_file"
> +test $? -ne 0 && error "failed to archive qemu"
>  for sm in $submodules; do
> -    git submodule update --init $sm
> -    test $? -ne 0 && error "failed to init submodule $sm"
> +	git submodule update --init "$sm"

The current archive-source.sh code is careful to not touch the
developer's current git checkout, because it runs  submodule
update in a clone'd tree. The earlier patch hunk gets rid of the
clone, so now submodule update is touching the developer's master
git checkout. IMHO this is not acceptable - running tests (which
use archive-source.sh) must never alter the developer's git checkout
status in any way.

The alternate patch mentioned above does that.

Even if we keep the git clone part, we could still usefully
switch to running git archive, instead of re-inventing it
via git ls-files + tar.

> +	test $? -ne 0 && error "failed to update submodule $sm"
> +	(cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"
> +	test $? -ne 0 && error "failed to archive submodule $sm"
> +	tar --concatenate --file "$tar_file" "$sub_file"
> +	test $? -ne 0 && error "failed append submodule $sm to $tar_file"
>  done
> -
> -if test -n "$submodules"; then
> -    {
> -        git ls-files || error "git ls-files failed"
> -        for sm in $submodules; do
> -            (cd $sm; git ls-files) | sed "s:^:$sm/:"
> -            if test "${PIPESTATUS[*]}" != "0 0"; then
> -                error "git ls-files in submodule $sm failed"
> -            fi
> -        done
> -    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
> -else
> -    git ls-files > "$list_file"
> -fi
> -
> -if test $? -ne 0; then
> -    error "failed to generate list file"
> -fi
> -
> -tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
> -
> -exit 0
> -- 
> 2.9.3
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH] scripts: use git archive in archive-source

[Gerd Hoffmann]

  Hi,

> > +git archive --format tar $HEAD > "$tar_file"
> > +test $? -ne 0 && error "failed to archive qemu"
> >  for sm in $submodules; do
> > -    git submodule update --init $sm
> > -    test $? -ne 0 && error "failed to init submodule $sm"
> > +	git submodule update --init "$sm"
> 
> The current archive-source.sh code is careful to not touch the
> developer's current git checkout, because it runs  submodule
> update in a clone'd tree. The earlier patch hunk gets rid of the
> clone, so now submodule update is touching the developer's master
> git checkout. IMHO this is not acceptable - running tests (which
> use archive-source.sh) must never alter the developer's git checkout
> status in any way.

Can easily be fixed, the sha1 can be pulled out of "git submodule
status" instead of updating the submodules & use HEAD.

The only question is what to do best on uninitialized submodules.
Options are:
 * fetch the submodule into the developer tree (once).
 * clone the submodule into a temporary directory (each run).
 * throw an error.

cheers,
  Gerd

Re: [Qemu-devel] [PATCH] scripts: use git archive in archive-source

[Daniel P. Berrangé]

On Thu, Jan 31, 2019 at 08:41:43AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > +git archive --format tar $HEAD > "$tar_file"
> > > +test $? -ne 0 && error "failed to archive qemu"
> > >  for sm in $submodules; do
> > > -    git submodule update --init $sm
> > > -    test $? -ne 0 && error "failed to init submodule $sm"
> > > +	git submodule update --init "$sm"
> > 
> > The current archive-source.sh code is careful to not touch the
> > developer's current git checkout, because it runs  submodule
> > update in a clone'd tree. The earlier patch hunk gets rid of the
> > clone, so now submodule update is touching the developer's master
> > git checkout. IMHO this is not acceptable - running tests (which
> > use archive-source.sh) must never alter the developer's git checkout
> > status in any way.
> 
> Can easily be fixed, the sha1 can be pulled out of "git submodule
> status" instead of updating the submodules & use HEAD.
> 
> The only question is what to do best on uninitialized submodules.
> Options are:
>  * fetch the submodule into the developer tree (once).

As above, we shouldn't ever touch the developer's git state.

>  * clone the submodule into a temporary directory (each run).

IMHO this is the only option

>  * throw an error.

Definitely don't want this, as an uninitialized submodule is going to
be a common case for the 'dtc' module when people have the suitable
version already on their host, so this would break the CI scripts.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH] scripts: use git archive in archive-source

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190130065530.6371-1-kraxel@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH] scripts: use git archive in archive-source
Type: series
Message-id: 20190130065530.6371-1-kraxel@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
42fe2deeca scripts: use git archive in archive-source

=== OUTPUT BEGIN ===
ERROR: code indent should never use tabs
#56: FILE: scripts/archive-source.sh:41:
+^Igit submodule update --init "$sm"$

ERROR: code indent should never use tabs
#57: FILE: scripts/archive-source.sh:42:
+^Itest $? -ne 0 && error "failed to update submodule $sm"$

ERROR: code indent should never use tabs
#58: FILE: scripts/archive-source.sh:43:
+^I(cd $sm; git archive --format tar --prefix "$sm/" HEAD) > "$sub_file"$

ERROR: code indent should never use tabs
#59: FILE: scripts/archive-source.sh:44:
+^Itest $? -ne 0 && error "failed to archive submodule $sm"$

ERROR: code indent should never use tabs
#60: FILE: scripts/archive-source.sh:45:
+^Itar --concatenate --file "$tar_file" "$sub_file"$

ERROR: code indent should never use tabs
#61: FILE: scripts/archive-source.sh:46:
+^Itest $? -ne 0 && error "failed append submodule $sm to $tar_file"$

total: 6 errors, 0 warnings, 41 lines checked

Commit 42fe2deecad0 (scripts: use git archive in archive-source) has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190130065530.6371-1-kraxel@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com