* Re: [PATCH] tests: docker: support mxe-based mingw builds
2020-08-19 8:02 [PATCH] tests: docker: support mxe-based mingw builds Paolo Bonzini
@ 2020-08-19 8:13 ` Marc-André Lureau
2020-08-19 8:20 ` Daniel P. Berrangé
2020-08-19 8:59 ` no-reply
2 siblings, 0 replies; 7+ messages in thread
From: Marc-André Lureau @ 2020-08-19 8:13 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: Peter Maydell, Alex Bennée, QEMU
[-- Attachment #1: Type: text/plain, Size: 4640 bytes --]
Hi
On Wed, Aug 19, 2020 at 12:03 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
> This can be run with docker-test-mingw@ubuntu1804-mxe, and is the setup
> that Peter uses to test cross-compilation.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>
I wish I had this a few times, thanks!
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
> tests/docker/common.rc | 18 ++++++-
> .../docker/dockerfiles/ubuntu1804-mxe.docker | 54 +++++++++++++++++++
> tests/docker/test-mingw | 11 +++-
> 3 files changed, 80 insertions(+), 3 deletions(-)
> create mode 100644 tests/docker/dockerfiles/ubuntu1804-mxe.docker
>
> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
> index ebc5b97ecf..402f6603b6 100755
> --- a/tests/docker/common.rc
> +++ b/tests/docker/common.rc
> @@ -15,10 +15,26 @@
> # overriden by TARGET_LIST if the user sets it.
> DEF_TARGET_LIST=${DEF_TARGET_LIST:-"x86_64-softmmu,aarch64-softmmu"}
>
> +has()
> +{
> + echo "$FEATURES" | grep -wq -e "$1"
> +}
> +
> +requires_any()
> +{
> + for c in $@; do
> + if has "$c"; then
> + return
> + fi
> + done
> + echo "None of prerequisites '$*' is present, skip"
> + exit 0
> +}
> +
> requires()
> {
> for c in $@; do
> - if ! echo "$FEATURES" | grep -wq -e "$c"; then
> + if ! has "$c"; then
> echo "Prerequisite '$c' not present, skip"
> exit 0
> fi
> diff --git a/tests/docker/dockerfiles/ubuntu1804-mxe.docker
> b/tests/docker/dockerfiles/ubuntu1804-mxe.docker
> new file mode 100644
> index 0000000000..91895db80d
> --- /dev/null
> +++ b/tests/docker/dockerfiles/ubuntu1804-mxe.docker
> @@ -0,0 +1,54 @@
> +FROM ubuntu:18.04
> +ENV PACKAGES \
> + ccache \
> + gcc \
> + gettext \
> + git \
> + gnupg \
> + gnupg2 \
> + make \
> + nsis \
> + python3-yaml \
> + python3-sphinx \
> + python3-setuptools \
> + texinfo
> +RUN apt-get update && \
> + DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
> +
> +ENV MXE_PACKAGES \
> + mxe-i686-w64-mingw32.shared-bzip2 \
> + mxe-i686-w64-mingw32.shared-curl \
> + mxe-i686-w64-mingw32.shared-glib \
> + mxe-i686-w64-mingw32.shared-gcc \
> + mxe-i686-w64-mingw32.shared-glib \
> + mxe-i686-w64-mingw32.shared-gmp \
> + mxe-i686-w64-mingw32.shared-gnutls \
> + mxe-i686-w64-mingw32.shared-gtk3 \
> + mxe-i686-w64-mingw32.shared-libjpeg-turbo \
> + mxe-i686-w64-mingw32.shared-libpng \
> + mxe-i686-w64-mingw32.shared-nettle \
> + mxe-i686-w64-mingw32.shared-nsis \
> + mxe-i686-w64-mingw32.shared-pixman \
> + mxe-i686-w64-mingw32.shared-pkgconf \
> + mxe-i686-w64-mingw32.shared-sdl2 \
> + mxe-x86-64-w64-mingw32.shared-bzip2 \
> + mxe-x86-64-w64-mingw32.shared-curl \
> + mxe-x86-64-w64-mingw32.shared-gcc \
> + mxe-x86-64-w64-mingw32.shared-glib \
> + mxe-x86-64-w64-mingw32.shared-gmp \
> + mxe-x86-64-w64-mingw32.shared-gnutls \
> + mxe-x86-64-w64-mingw32.shared-gtk3 \
> + mxe-x86-64-w64-mingw32.shared-libjpeg-turbo \
> + mxe-x86-64-w64-mingw32.shared-libpng \
> + mxe-x86-64-w64-mingw32.shared-nettle \
> + mxe-x86-64-w64-mingw32.shared-nsis \
> + mxe-x86-64-w64-mingw32.shared-pixman \
> + mxe-x86-64-w64-mingw32.shared-pkgconf \
> + mxe-x86-64-w64-mingw32.shared-sdl2
> +
> +RUN echo "deb http://pkg.mxe.cc/repos/apt bionic main" > \
> + /etc/apt/sources.list.d/mxeapt.list && \
> + apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
> C6BF758A33A3A276 && \
> + apt-get update && \
> + DEBIAN_FRONTEND=noninteractive apt-get install -y $MXE_PACKAGES
> +ENV FEATURES mxe
> diff --git a/tests/docker/test-mingw b/tests/docker/test-mingw
> index c30eb654eb..9e2fadb11a 100755
> --- a/tests/docker/test-mingw
> +++ b/tests/docker/test-mingw
> @@ -13,11 +13,18 @@
>
> . common.rc
>
> -requires mingw dtc
> +requires dtc
> +requires_any mingw mxe
>
> cd "$BUILD_DIR"
>
> -for prefix in x86_64-w64-mingw32- i686-w64-mingw32-; do
> +if has mingw; then
> + prefixes='x86_64-w64-mingw32- i686-w64-mingw32-'
> +else
> + prefixes='x86_64-w64-mingw32.shared- i686-w64-mingw32.shared-'
> + export PATH=/usr/lib/mxe/usr/bin:$PATH
> +fi
> +for prefix in $prefixes; do
> TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
> build_qemu --cross-prefix=$prefix \
> --enable-trace-backends=simple \
> --
> 2.26.2
>
>
>
--
Marc-André Lureau
[-- Attachment #2: Type: text/html, Size: 6121 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] tests: docker: support mxe-based mingw builds
2020-08-19 8:02 [PATCH] tests: docker: support mxe-based mingw builds Paolo Bonzini
2020-08-19 8:13 ` Marc-André Lureau
@ 2020-08-19 8:20 ` Daniel P. Berrangé
2020-08-19 10:42 ` Paolo Bonzini
2020-08-19 8:59 ` no-reply
2 siblings, 1 reply; 7+ messages in thread
From: Daniel P. Berrangé @ 2020-08-19 8:20 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: peter.maydell, alex.bennee, qemu-devel
On Wed, Aug 19, 2020 at 04:02:06AM -0400, Paolo Bonzini wrote:
> This can be run with docker-test-mingw@ubuntu1804-mxe, and is the setup
> that Peter uses to test cross-compilation.
We already have docker containers with MXE based on Debian:
debian-win32-cross.docker
debian-win64-cross.docker
your image uses a different naming convention, and puts both
32 and 64 bit in the same image.
I feel like we should have the Ubuntu variant follow the same
structure and naming as the Debian variant for consistency.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> tests/docker/common.rc | 18 ++++++-
> .../docker/dockerfiles/ubuntu1804-mxe.docker | 54 +++++++++++++++++++
> tests/docker/test-mingw | 11 +++-
> 3 files changed, 80 insertions(+), 3 deletions(-)
> create mode 100644 tests/docker/dockerfiles/ubuntu1804-mxe.docker
>
> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
> index ebc5b97ecf..402f6603b6 100755
> --- a/tests/docker/common.rc
> +++ b/tests/docker/common.rc
> @@ -15,10 +15,26 @@
> # overriden by TARGET_LIST if the user sets it.
> DEF_TARGET_LIST=${DEF_TARGET_LIST:-"x86_64-softmmu,aarch64-softmmu"}
>
> +has()
> +{
> + echo "$FEATURES" | grep -wq -e "$1"
> +}
> +
> +requires_any()
> +{
> + for c in $@; do
> + if has "$c"; then
> + return
> + fi
> + done
> + echo "None of prerequisites '$*' is present, skip"
> + exit 0
> +}
> +
> requires()
> {
> for c in $@; do
> - if ! echo "$FEATURES" | grep -wq -e "$c"; then
> + if ! has "$c"; then
> echo "Prerequisite '$c' not present, skip"
> exit 0
> fi
> diff --git a/tests/docker/dockerfiles/ubuntu1804-mxe.docker b/tests/docker/dockerfiles/ubuntu1804-mxe.docker
> new file mode 100644
> index 0000000000..91895db80d
> --- /dev/null
> +++ b/tests/docker/dockerfiles/ubuntu1804-mxe.docker
> @@ -0,0 +1,54 @@
> +FROM ubuntu:18.04
> +ENV PACKAGES \
> + ccache \
> + gcc \
> + gettext \
> + git \
> + gnupg \
> + gnupg2 \
> + make \
> + nsis \
> + python3-yaml \
> + python3-sphinx \
> + python3-setuptools \
> + texinfo
> +RUN apt-get update && \
> + DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES
> +
> +ENV MXE_PACKAGES \
> + mxe-i686-w64-mingw32.shared-bzip2 \
> + mxe-i686-w64-mingw32.shared-curl \
> + mxe-i686-w64-mingw32.shared-glib \
> + mxe-i686-w64-mingw32.shared-gcc \
> + mxe-i686-w64-mingw32.shared-glib \
> + mxe-i686-w64-mingw32.shared-gmp \
> + mxe-i686-w64-mingw32.shared-gnutls \
> + mxe-i686-w64-mingw32.shared-gtk3 \
> + mxe-i686-w64-mingw32.shared-libjpeg-turbo \
> + mxe-i686-w64-mingw32.shared-libpng \
> + mxe-i686-w64-mingw32.shared-nettle \
> + mxe-i686-w64-mingw32.shared-nsis \
> + mxe-i686-w64-mingw32.shared-pixman \
> + mxe-i686-w64-mingw32.shared-pkgconf \
> + mxe-i686-w64-mingw32.shared-sdl2 \
> + mxe-x86-64-w64-mingw32.shared-bzip2 \
> + mxe-x86-64-w64-mingw32.shared-curl \
> + mxe-x86-64-w64-mingw32.shared-gcc \
> + mxe-x86-64-w64-mingw32.shared-glib \
> + mxe-x86-64-w64-mingw32.shared-gmp \
> + mxe-x86-64-w64-mingw32.shared-gnutls \
> + mxe-x86-64-w64-mingw32.shared-gtk3 \
> + mxe-x86-64-w64-mingw32.shared-libjpeg-turbo \
> + mxe-x86-64-w64-mingw32.shared-libpng \
> + mxe-x86-64-w64-mingw32.shared-nettle \
> + mxe-x86-64-w64-mingw32.shared-nsis \
> + mxe-x86-64-w64-mingw32.shared-pixman \
> + mxe-x86-64-w64-mingw32.shared-pkgconf \
> + mxe-x86-64-w64-mingw32.shared-sdl2
> +
> +RUN echo "deb http://pkg.mxe.cc/repos/apt bionic main" > \
> + /etc/apt/sources.list.d/mxeapt.list && \
> + apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C6BF758A33A3A276 && \
> + apt-get update && \
> + DEBIAN_FRONTEND=noninteractive apt-get install -y $MXE_PACKAGES
> +ENV FEATURES mxe
> diff --git a/tests/docker/test-mingw b/tests/docker/test-mingw
> index c30eb654eb..9e2fadb11a 100755
> --- a/tests/docker/test-mingw
> +++ b/tests/docker/test-mingw
> @@ -13,11 +13,18 @@
>
> . common.rc
>
> -requires mingw dtc
> +requires dtc
> +requires_any mingw mxe
>
> cd "$BUILD_DIR"
>
> -for prefix in x86_64-w64-mingw32- i686-w64-mingw32-; do
> +if has mingw; then
> + prefixes='x86_64-w64-mingw32- i686-w64-mingw32-'
> +else
> + prefixes='x86_64-w64-mingw32.shared- i686-w64-mingw32.shared-'
> + export PATH=/usr/lib/mxe/usr/bin:$PATH
> +fi
> +for prefix in $prefixes; do
> TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
> build_qemu --cross-prefix=$prefix \
> --enable-trace-backends=simple \
> --
> 2.26.2
>
>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] tests: docker: support mxe-based mingw builds
2020-08-19 8:20 ` Daniel P. Berrangé
@ 2020-08-19 10:42 ` Paolo Bonzini
2020-08-19 11:09 ` Daniel P. Berrangé
0 siblings, 1 reply; 7+ messages in thread
From: Paolo Bonzini @ 2020-08-19 10:42 UTC (permalink / raw)
To: Daniel P. Berrangé; +Cc: peter.maydell, alex.bennee, qemu-devel
On 19/08/20 10:20, Daniel P. Berrangé wrote:
> We already have docker containers with MXE based on Debian:
>
> debian-win32-cross.docker
> debian-win64-cross.docker
>
> your image uses a different naming convention, and puts both
> 32 and 64 bit in the same image.
Yeah, that's what test-mingw expects.
> I feel like we should have the Ubuntu variant follow the same
> structure and naming as the Debian variant for consistency.
My patch follows the Fedora variant, so that test-mingw runs. That
ensures that NSIS is covered as well. One possibility could be:
- create fedora-win*-cross dockerfiles
- add ENV FEATURES $FEATURES mingw to the win*-cross dockerfiles
- look for the feature in test-full and test-quick, and run "make
installer" if so.
- drop test-mingw completely, and adjust Patchew to use
docker-test-quick@fedora-win{32,64}-cross instead
Paolo
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] tests: docker: support mxe-based mingw builds
2020-08-19 10:42 ` Paolo Bonzini
@ 2020-08-19 11:09 ` Daniel P. Berrangé
2020-08-19 11:41 ` Paolo Bonzini
0 siblings, 1 reply; 7+ messages in thread
From: Daniel P. Berrangé @ 2020-08-19 11:09 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: peter.maydell, alex.bennee, qemu-devel
On Wed, Aug 19, 2020 at 12:42:44PM +0200, Paolo Bonzini wrote:
> On 19/08/20 10:20, Daniel P. Berrangé wrote:
> > We already have docker containers with MXE based on Debian:
> >
> > debian-win32-cross.docker
> > debian-win64-cross.docker
> >
> > your image uses a different naming convention, and puts both
> > 32 and 64 bit in the same image.
>
> Yeah, that's what test-mingw expects.
Ah, I missed that.
>
> > I feel like we should have the Ubuntu variant follow the same
> > structure and naming as the Debian variant for consistency.
>
> My patch follows the Fedora variant, so that test-mingw runs. That
> ensures that NSIS is covered as well. One possibility could be:
>
> - create fedora-win*-cross dockerfiles
Yeah, I think that'd make sense, as it'd enable a simple trick we
do in libvirt.
In all the dockerfile recipe we set an env
ENV MESON_OPTS "--cross-file=/usr/share/mingw/toolchain-mingw32.meson"
And in other linux-cross builds, we do similar
ENV MESON_OPTS "--cross-file=i686-linux-gnu"
So now from host side can just invoke "meson $MESON_OPTS" and it will
do the right thing according to whatever the container image was
installed with, regardless of whether it is a cross, or native build.
This obviously only works if you have separate images for win32
and win64.
> - add ENV FEATURES $FEATURES mingw to the win*-cross dockerfiles
>
> - look for the feature in test-full and test-quick, and run "make
> installer" if so.
I'd suggest that "make installer" should be a part of "make" not
a separate thing that needs running manually.
eg if we're configure'ing for mingw, configure should check whether
we have the NSIS tools available and if so, then enable NSIS as a
standard build output. We could have a configure option to enable/disable
NSIS explicitly.
This eliminates the second bit of special casing for mingw
> - drop test-mingw completely, and adjust Patchew to use
> docker-test-quick@fedora-win{32,64}-cross instead
Yes, dropping test-mingw would be better, but I'm not sure you can
use 'test-quick' as that runs unit tests which would require wine
to be present. 'test-build' would be closer to what test-mingw
does.
Anyway, if the images and make rules are created & setup in a suitable
way, there should be no need to do anything special in the host side
for cross-builds - just pick the image you want and it should "just work".
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] tests: docker: support mxe-based mingw builds
2020-08-19 11:09 ` Daniel P. Berrangé
@ 2020-08-19 11:41 ` Paolo Bonzini
0 siblings, 0 replies; 7+ messages in thread
From: Paolo Bonzini @ 2020-08-19 11:41 UTC (permalink / raw)
To: Daniel P. Berrangé; +Cc: peter.maydell, alex.bennee, qemu-devel
On 19/08/20 13:09, Daniel P. Berrangé wrote:
> On Wed, Aug 19, 2020 at 12:42:44PM +0200, Paolo Bonzini wrote:
>> On 19/08/20 10:20, Daniel P. Berrangé wrote:
>>> We already have docker containers with MXE based on Debian:
>>>
>>> debian-win32-cross.docker
>>> debian-win64-cross.docker
>>>
>>> your image uses a different naming convention, and puts both
>>> 32 and 64 bit in the same image.
>>
>> Yeah, that's what test-mingw expects.
>
> Ah, I missed that.
>
>>
>>> I feel like we should have the Ubuntu variant follow the same
>>> structure and naming as the Debian variant for consistency.
>>
>> My patch follows the Fedora variant, so that test-mingw runs. That
>> ensures that NSIS is covered as well. One possibility could be:
>>
>> - create fedora-win*-cross dockerfiles
>
> Yeah, I think that'd make sense, as it'd enable a simple trick we
> do in libvirt.
>
> In all the dockerfile recipe we set an env
>
> ENV MESON_OPTS "--cross-file=/usr/share/mingw/toolchain-mingw32.meson"
>
> And in other linux-cross builds, we do similar
>
> ENV MESON_OPTS "--cross-file=i686-linux-gnu"
Yeah, that's the same as QEMU's QEMU_CONFIGURE_OPTS.
>> - add ENV FEATURES $FEATURES mingw to the win*-cross dockerfiles
>>
>> - look for the feature in test-full and test-quick, and run "make
>> installer" if so.
>
> I'd suggest that "make installer" should be a part of "make" not
> a separate thing that needs running manually.
>
> eg if we're configure'ing for mingw, configure should check whether
> we have the NSIS tools available and if so, then enable NSIS as a
> standard build output. We could have a configure option to enable/disable
> NSIS explicitly.
>
> This eliminates the second bit of special casing for mingw
Yeah, that might be an idea. I'm not sure if it would work however
because "make installer" invokes "make install", so there could be
recursive (and non-reentrant) invocations.
Paolo
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] tests: docker: support mxe-based mingw builds
2020-08-19 8:02 [PATCH] tests: docker: support mxe-based mingw builds Paolo Bonzini
2020-08-19 8:13 ` Marc-André Lureau
2020-08-19 8:20 ` Daniel P. Berrangé
@ 2020-08-19 8:59 ` no-reply
2 siblings, 0 replies; 7+ messages in thread
From: no-reply @ 2020-08-19 8:59 UTC (permalink / raw)
To: pbonzini; +Cc: peter.maydell, alex.bennee, qemu-devel
Patchew URL: https://patchew.org/QEMU/20200819080206.27423-1-pbonzini@redhat.com/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===
TEST check-unit: tests/test-char
Unexpected error in object_property_try_add() at /tmp/qemu-test/src/qom/object.c:1181:
attempt to add duplicate property 'serial-id' to object (type 'container')
ERROR test-char - too few tests run (expected 38, got 9)
make: *** [check-unit] Error 1
make: *** Waiting for unfinished jobs....
TEST check-qtest-x86_64: tests/qtest/hd-geo-test
qemu-system-aarch64: -accel kvm: invalid accelerator kvm
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=238b70bd18a14cdabb8a765c46c87ecc', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-oytm7a2n/src/docker-src.2020-08-19-04.46.55.6123:/var/tmp/qemu:z,ro', 'qemu/centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=238b70bd18a14cdabb8a765c46c87ecc
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-oytm7a2n/src'
make: *** [docker-run-test-quick@centos7] Error 2
real 12m56.910s
user 0m9.512s
The full log is available at
http://patchew.org/logs/20200819080206.27423-1-pbonzini@redhat.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
^ permalink raw reply [flat|nested] 7+ messages in thread