[PATCH] mktarball: For qemu upstream, use their scripts/archive-source.sh

[PATCH] mktarball: For qemu upstream, use their scripts/archive-source.sh

[Ian Jackson]

qemu upstream uses git submodules.  git archive does not work with git
submodules (and could not work properly with them, because this is one
of the many things it is inherently impossible to do correctly with
git submodules).

qemu upstream have worked around this by providing a rather scary
shell script which attempts to do roughly the right thing.  It's close
enough that we can use it with only minor precautions.

Unfortunately this does mean that `mktarball' now executes the qemu
source code it was using, rather than merely shuffling it about, as it
did previously.  I think this is a less bad ill than copying (and,
effectively, forking) the scary script.

CC: Wei Liu <wei.liu2@citrix.com>
CC: George Dunlap <george.dunlap@eu.citrix.com>
CC: Juergen Gross <jgross@suse.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 tools/misc/mktarball | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/tools/misc/mktarball b/tools/misc/mktarball
index 73282b5..42d5430 100755
--- a/tools/misc/mktarball
+++ b/tools/misc/mktarball
@@ -29,7 +29,21 @@ mkdir -p $tdir
 
 git_archive_into $xen_root $tdir/xen-$desc
 
-git_archive_into $xen_root/tools/qemu-xen-dir-remote $tdir/xen-$desc/tools/qemu-xen
+# We can't use git_archive_into with qemu upstream because it uses
+# git-submodules.  git-submodules are an inherently broken git feature
+# which should never be used in any circumstance.  Unfortunately, qemu
+# upstream uses them.  Relevantly for us, git archive does not work
+# properly when there are submodules.
+(
+    cd $xen_root/tools/qemu-xen-dir-remote
+    # if it's not clean, the qemu script will call `git stash' !
+    git --no-pager diff --stat HEAD
+    scripts/archive-source.sh $tdir/xen-$desc/tools/qemu-xen.tar
+    cd $tdir/xen-$desc/tools
+    mkdir qemu-xen
+    tar <qemu-xen.tar Cxf qemu-xen -
+    rm qemu-xen.tar
+)
 
 git_archive_into $xen_root/tools/qemu-xen-traditional-dir-remote $tdir/xen-$desc/tools/qemu-xen-traditional
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] mktarball: For qemu upstream, use their scripts/archive-source.sh

[George Dunlap]

On 04/17/2018 06:04 PM, Ian Jackson wrote:
> qemu upstream uses git submodules.  git archive does not work with git
> submodules (and could not work properly with them, because this is one
> of the many things it is inherently impossible to do correctly with
> git submodules).
> 
> qemu upstream have worked around this by providing a rather scary
> shell script which attempts to do roughly the right thing.  It's close
> enough that we can use it with only minor precautions.
> 
> Unfortunately this does mean that `mktarball' now executes the qemu
> source code it was using, rather than merely shuffling it about, as it
> did previously.  I think this is a less bad ill than copying (and,
> effectively, forking) the scary script.
> 
> CC: Wei Liu <wei.liu2@citrix.com>
> CC: George Dunlap <george.dunlap@eu.citrix.com>
> CC: Juergen Gross <jgross@suse.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  tools/misc/mktarball | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/misc/mktarball b/tools/misc/mktarball
> index 73282b5..42d5430 100755
> --- a/tools/misc/mktarball
> +++ b/tools/misc/mktarball
> @@ -29,7 +29,21 @@ mkdir -p $tdir
>  
>  git_archive_into $xen_root $tdir/xen-$desc
>  
> -git_archive_into $xen_root/tools/qemu-xen-dir-remote $tdir/xen-$desc/tools/qemu-xen
> +# We can't use git_archive_into with qemu upstream because it uses
> +# git-submodules.  git-submodules are an inherently broken git feature
> +# which should never be used in any circumstance.  Unfortunately, qemu
> +# upstream uses them.  Relevantly for us, git archive does not work
> +# properly when there are submodules.
> +(
> +    cd $xen_root/tools/qemu-xen-dir-remote
> +    # if it's not clean, the qemu script will call `git stash' !
> +    git --no-pager diff --stat HEAD
> +    scripts/archive-source.sh $tdir/xen-$desc/tools/qemu-xen.tar
> +    cd $tdir/xen-$desc/tools
> +    mkdir qemu-xen
> +    tar <qemu-xen.tar Cxf qemu-xen -

I'm not sure why we're piping from a file and then specifying `-`, but
now's not really the time for bike shedding:

Acked-by: George Dunlap <george.dunlap@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] mktarball: For qemu upstream, use their scripts/archive-source.sh

[Ian Jackson]

George Dunlap writes ("Re: [PATCH] mktarball: For qemu upstream, use their scripts/archive-source.sh"):
> On 04/17/2018 06:04 PM, Ian Jackson wrote:
> > +    tar <qemu-xen.tar Cxf qemu-xen -
> 
> I'm not sure why we're piping from a file and then specifying `-`, but
> now's not really the time for bike shedding:

C and f interact in a way that I find odd, so I prefer to use <.

> Acked-by: George Dunlap <george.dunlap@citrix.com>

Thanks.  Now in 4.11.0-rc1 and in staging.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] mktarball: For qemu upstream, use their scripts/archive-source.sh

[Anthony PERARD]

On Tue, Apr 17, 2018 at 06:04:37PM +0100, Ian Jackson wrote:
> diff --git a/tools/misc/mktarball b/tools/misc/mktarball
> index 73282b5..42d5430 100755
> --- a/tools/misc/mktarball
> +++ b/tools/misc/mktarball
> @@ -29,7 +29,21 @@ mkdir -p $tdir
>  
>  git_archive_into $xen_root $tdir/xen-$desc
>  
> -git_archive_into $xen_root/tools/qemu-xen-dir-remote $tdir/xen-$desc/tools/qemu-xen
> +# We can't use git_archive_into with qemu upstream because it uses
> +# git-submodules.  git-submodules are an inherently broken git feature
> +# which should never be used in any circumstance.  Unfortunately, qemu

I don't think xen.git is the right place to bash the use of git
submodule. Especially when there is no way, in xen.git, to find out
ahead of time which git trees the xen build system is going to want to
download.

> +# upstream uses them.  Relevantly for us, git archive does not work
> +# properly when there are submodules.
> +(
> +    cd $xen_root/tools/qemu-xen-dir-remote
> +    # if it's not clean, the qemu script will call `git stash' !

It's actually `git stash create`. The script doesn't modify the source
worktree. The script seems to be intended to make a tarball of the
worktree (so also things not commited yet), and we use it make a tarball
of a commit.

There is scripts/make-release that is simpler (called via `make dist`),
but it probably stuff the tarball with more than we need (roms).

> +    git --no-pager diff --stat HEAD
> +    scripts/archive-source.sh $tdir/xen-$desc/tools/qemu-xen.tar
> +    cd $tdir/xen-$desc/tools
> +    mkdir qemu-xen
> +    tar <qemu-xen.tar Cxf qemu-xen -
> +    rm qemu-xen.tar
> +)

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel