[Qemu-devel] [PATCH 0/4] docker: Improve 'host-cross' image

[Qemu-devel] [PATCH 0/4] docker: Improve 'host-cross' image

[Philippe Mathieu-Daudé]

Sorry I don't have a better name for this image,
it is meant to cross-build for the same architecture
than the host.

Due to the bad name, explicitely tagged the last 2
patches as 'RFC'.

Tested on x86_64 and aarch64, should also work on
s390x and ppc64le (per the Docker doc).

If they are volunteers for test, the command is as simple as:

  $ make docker-test-quick@debian-host-cross [V=1]

Regards,

Phil.

Philippe Mathieu-Daudé (4):
  docker: Use the host architecture
  docker: Improve how the virgl library is built
  docker: Rename 'amd64' image as 'host-cross'
  docker: Add many libraries to the 'host-cross' image

 .shippable.yml                                |  4 +-
 tests/docker/Makefile.include                 |  2 +-
 tests/docker/dockerfiles/debian-amd64.docker  | 39 ---------
 .../dockerfiles/debian-host-cross.docker      | 83 +++++++++++++++++++
 4 files changed, 86 insertions(+), 42 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-amd64.docker
 create mode 100644 tests/docker/dockerfiles/debian-host-cross.docker

-- 
2.18.0

[Qemu-devel] [PATCH 1/4] docker: Use the host architecture

[Philippe Mathieu-Daudé]

This image is badly named and not amd64 specific.

Use `dpkg --print-architecture` to get the host architecture.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-amd64.docker | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index eb13f06ed1..3312374304 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -30,9 +30,11 @@ RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --with-glx --disabl
 # netmap
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
     apt-get install -y --no-install-recommends \
-        linux-headers-amd64
+        linux-headers-$(dpkg --print-architecture)
 RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
-RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
+RUN cd /usr/src/netmap/LINUX && \
+    ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-$(dpkg --print-architecture)) && \
+    make install
 ENV QEMU_CONFIGURE_OPTS --enable-netmap
 
 # gcrypt
-- 
2.18.0

[Qemu-devel] [PATCH 2/4] docker: Improve how the virgl library is built

[Philippe Mathieu-Daudé]

- Call autogen once in the same layer of the git repository clone,
- Disable build warnings (they are not relevant for QEMU),
- Call ldconfig once installed.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-amd64.docker | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index 3312374304..d79a599a0d 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -24,8 +24,12 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
         libegl1-mesa-dev \
         libepoxy-dev \
         libgbm-dev
-RUN git clone git://anongit.freedesktop.org/virglrenderer /usr/src/virglrenderer
-RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --with-glx --disable-tests && make install
+RUN git clone git://anongit.freedesktop.org/virglrenderer /usr/src/virglrenderer && \
+    cd /usr/src/virglrenderer && ./autogen.sh
+RUN cd /usr/src/virglrenderer && \
+    CPPFLAGS=-w ./configure --with-glx --disable-tests && \
+    make install && \
+    ldconfig
 
 # netmap
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
-- 
2.18.0

[Qemu-devel] [RFC PATCH 3/4] docker: Rename 'amd64' image as 'host-cross'

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml                                           | 4 ++--
 tests/docker/Makefile.include                            | 2 +-
 .../{debian-amd64.docker => debian-host-cross.docker}    | 9 ++++++---
 3 files changed, 9 insertions(+), 6 deletions(-)
 rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host-cross.docker} (82%)

diff --git a/.shippable.yml b/.shippable.yml
index f74a3de3ff..7625ba8f27 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -5,8 +5,8 @@ env:
   global:
     - LC_ALL=C
   matrix:
-    - IMAGE=debian-amd64
-      TARGET_LIST=x86_64-softmmu,x86_64-linux-user
+    - IMAGE=debian-host-cross
+      TARGET_LIST=aarch64-softmmu,x86_64-softmmu,x86_64-linux-user
     - IMAGE=debian-win32-cross
       TARGET_LIST=arm-softmmu,i386-softmmu,lm32-softmmu
     - IMAGE=debian-win64-cross
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index b2a7e761cc..2db4190b4c 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -83,7 +83,7 @@ endif
 # Enforce dependencies for composite images
 docker-image-debian: docker-image-debian9
 docker-image-debian8-mxe: docker-image-debian8
-docker-image-debian-amd64: docker-image-debian9
+docker-image-debian-host-cross: docker-image-debian9
 docker-image-debian-armel-cross: docker-image-debian9
 docker-image-debian-armhf-cross: docker-image-debian9
 docker-image-debian-arm64-cross: docker-image-debian9
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-host-cross.docker
similarity index 82%
rename from tests/docker/dockerfiles/debian-amd64.docker
rename to tests/docker/dockerfiles/debian-host-cross.docker
index d79a599a0d..17d937efc9 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-host-cross.docker
@@ -1,8 +1,11 @@
 #
-# Docker x86_64 target
+# Docker 'host' target (same architecture than the host)
 #
-# This docker target builds on the debian Stretch base image. Further
-# libraries which are not widely available are installed by hand.
+# This docker target builds on the debian Stretch base image.
+#
+# It is intented to build the host architecture, with as much
+# features as possible (libraries which are not widely available
+# are installed by hand).
 #
 FROM qemu:debian9
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
-- 
2.18.0

[Qemu-devel] [RFC PATCH 4/4] docker: Add many libraries to the 'host-cross' image

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
To avoid some problem with Docker permissions on NUMA systems,
the libnuma-dev package is deliberately not installed (yet).
---
 .../dockerfiles/debian-host-cross.docker      | 37 ++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-host-cross.docker b/tests/docker/dockerfiles/debian-host-cross.docker
index 17d937efc9..89ca0a4daf 100644
--- a/tests/docker/dockerfiles/debian-host-cross.docker
+++ b/tests/docker/dockerfiles/debian-host-cross.docker
@@ -15,11 +15,46 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
     apt-get install -y --no-install-recommends \
+        libaio-dev \
+        libasound2-dev \
+        libattr1-dev \
+        libbluetooth-dev \
+        libbrlapi-dev \
         libbz2-dev \
+        libcacard-dev \
+        libcap-dev \
+        libcap-ng-dev \
+        libcapstone-dev \
+        libcurl4-gnutls-dev \
+        libepoxy-dev \
+        libfdt-dev \
+        libibverbs-dev \
+        libiscsi-dev \
+        libjemalloc-dev \
+        libjpeg-dev \
+        liblttng-ust-dev \
         liblzo2-dev \
+        libncursesw5-dev \
+        libnfs-dev \
+        libnss3-dev \
+        libpixman-1-dev \
+        libpulse-dev \
+        librados-dev \
+        librbd-dev \
         librdmacm-dev \
+        libseccomp-dev \
         libsnappy-dev \
-        libvte-dev
+        libspice-protocol-dev \
+        libssh2-1-dev \
+        liburcu-dev \
+        libusb-1.0-0-dev \
+        libusbredirhost-dev \
+        libusbredirparser-dev \
+        libvdeplug-dev \
+        libvte-dev \
+        libxen-dev \
+        uuid-dev \
+        xfslibs-dev
 
 # virgl
 RUN DEBIAN_FRONTEND=noninteractive eatmydata \
-- 
2.18.0

Re: [Qemu-devel] [PATCH 0/4] docker: Improve 'host-cross' image

[Cornelia Huck]

On Mon, 16 Jul 2018 23:29:20 -0300
Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:

> Sorry I don't have a better name for this image,
> it is meant to cross-build for the same architecture
> than the host.
> 
> Due to the bad name, explicitely tagged the last 2
> patches as 'RFC'.
> 
> Tested on x86_64 and aarch64, should also work on
> s390x and ppc64le (per the Docker doc).
> 
> If they are volunteers for test, the command is as simple as:
> 
>   $ make docker-test-quick@debian-host-cross [V=1]

On s390x, all looks reasonable until it tries to install libxen-dev:

Step 4/14 : RUN DEBIAN_FRONTEND=noninteractive eatmydata     apt-get install -y --no-install-recommends         libaio-dev         libasound2-dev         libattr1-dev         libbluetooth-dev         libbrlapi-dev         libbz2-dev         libcacard-dev         libcap-dev         libcap-ng-dev         libcapstone-dev         libcurl4-gnutls-dev         libepoxy-dev         libfdt-dev         libibverbs-dev         libiscsi-dev         libjemalloc-dev         libjpeg-dev         liblttng-ust-dev         liblzo2-dev         libncursesw5-dev         libnfs-dev         libnss3-dev         libpixman-1-dev         libpulse-dev         librados-dev         librbd-dev         librdmacm-dev         libseccomp-dev         libsnappy-dev         libspice-protocol-dev         libssh2-1-dev         liburcu-dev         libusb-1.0-0-dev         libusbredirhost-dev         libusbredirparser-dev         libvdeplug-dev         libvte-dev         libxen-dev         uuid-dev         xfslibs-dev
 ---> Running in a7f6ae892a05
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libxen-dev

Re: [Qemu-devel] [PATCH 1/4] docker: Use the host architecture

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> This image is badly named and not amd64 specific.
>
> Use `dpkg --print-architecture` to get the host architecture.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

It's like you've been reading my mind ;-)

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>


> ---
>  tests/docker/dockerfiles/debian-amd64.docker | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
> index eb13f06ed1..3312374304 100644
> --- a/tests/docker/dockerfiles/debian-amd64.docker
> +++ b/tests/docker/dockerfiles/debian-amd64.docker
> @@ -30,9 +30,11 @@ RUN cd /usr/src/virglrenderer && ./autogen.sh && ./configure --with-glx --disabl
>  # netmap
>  RUN DEBIAN_FRONTEND=noninteractive eatmydata \
>      apt-get install -y --no-install-recommends \
> -        linux-headers-amd64
> +        linux-headers-$(dpkg --print-architecture)
>  RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap
> -RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install
> +RUN cd /usr/src/netmap/LINUX && \
> +    ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-$(dpkg --print-architecture)) && \
> +    make install
>  ENV QEMU_CONFIGURE_OPTS --enable-netmap
>
>  # gcrypt


--
Alex Bennée

Re: [Qemu-devel] [RFC PATCH 3/4] docker: Rename 'amd64' image as 'host-cross'

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .shippable.yml                                           | 4 ++--
>  tests/docker/Makefile.include                            | 2 +-
>  .../{debian-amd64.docker => debian-host-cross.docker}    | 9
> ++++++---

I'd prefer debian-amd64->debian-host as it's not really a cross
compiler.

Otherwise:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>


>  3 files changed, 9 insertions(+), 6 deletions(-)
>  rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host-cross.docker} (82%)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index f74a3de3ff..7625ba8f27 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -5,8 +5,8 @@ env:
>    global:
>      - LC_ALL=C
>    matrix:
> -    - IMAGE=debian-amd64
> -      TARGET_LIST=x86_64-softmmu,x86_64-linux-user
> +    - IMAGE=debian-host-cross
> +      TARGET_LIST=aarch64-softmmu,x86_64-softmmu,x86_64-linux-user
>      - IMAGE=debian-win32-cross
>        TARGET_LIST=arm-softmmu,i386-softmmu,lm32-softmmu
>      - IMAGE=debian-win64-cross
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index b2a7e761cc..2db4190b4c 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -83,7 +83,7 @@ endif
>  # Enforce dependencies for composite images
>  docker-image-debian: docker-image-debian9
>  docker-image-debian8-mxe: docker-image-debian8
> -docker-image-debian-amd64: docker-image-debian9
> +docker-image-debian-host-cross: docker-image-debian9
>  docker-image-debian-armel-cross: docker-image-debian9
>  docker-image-debian-armhf-cross: docker-image-debian9
>  docker-image-debian-arm64-cross: docker-image-debian9
> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-host-cross.docker
> similarity index 82%
> rename from tests/docker/dockerfiles/debian-amd64.docker
> rename to tests/docker/dockerfiles/debian-host-cross.docker
> index d79a599a0d..17d937efc9 100644
> --- a/tests/docker/dockerfiles/debian-amd64.docker
> +++ b/tests/docker/dockerfiles/debian-host-cross.docker
> @@ -1,8 +1,11 @@
>  #
> -# Docker x86_64 target
> +# Docker 'host' target (same architecture than the host)
>  #
> -# This docker target builds on the debian Stretch base image. Further
> -# libraries which are not widely available are installed by hand.
> +# This docker target builds on the debian Stretch base image.
> +#
> +# It is intented to build the host architecture, with as much
> +# features as possible (libraries which are not widely available
> +# are installed by hand).
>  #
>  FROM qemu:debian9
>  MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>


--
Alex Bennée

Re: [Qemu-devel] [PATCH 0/4] docker: Improve 'host-cross' image

[Philippe Mathieu-Daudé]

Hi Cornelia,

On 07/17/2018 11:12 AM, Cornelia Huck wrote:
> On Mon, 16 Jul 2018 23:29:20 -0300
> Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> 
>> Sorry I don't have a better name for this image,
>> it is meant to cross-build for the same architecture
>> than the host.
>>
>> Due to the bad name, explicitely tagged the last 2
>> patches as 'RFC'.
>>
>> Tested on x86_64 and aarch64, should also work on
>> s390x and ppc64le (per the Docker doc).
>>
>> If they are volunteers for test, the command is as simple as:
>>
>>   $ make docker-test-quick@debian-host-cross [V=1]
> 
> On s390x, all looks reasonable until it tries to install libxen-dev:

https://packages.debian.org/stretch/libxen-dev#pdownload
Oh, no Xen for s390x nor ppc64el... OK.

> Step 4/14 : RUN DEBIAN_FRONTEND=noninteractive eatmydata     apt-get install -y --no-install-recommends         libaio-dev         libasound2-dev         libattr1-dev         libbluetooth-dev         libbrlapi-dev         libbz2-dev         libcacard-dev         libcap-dev         libcap-ng-dev         libcapstone-dev         libcurl4-gnutls-dev         libepoxy-dev         libfdt-dev         libibverbs-dev         libiscsi-dev         libjemalloc-dev         libjpeg-dev         liblttng-ust-dev         liblzo2-dev         libncursesw5-dev         libnfs-dev         libnss3-dev         libpixman-1-dev         libpulse-dev         librados-dev         librbd-dev         librdmacm-dev         libseccomp-dev         libsnappy-dev         libspice-protocol-dev         libssh2-1-dev         liburcu-dev         libusb-1.0-0-dev         libusbredirhost-dev         libusbredirparser-dev         libvdeplug-dev         libvte-dev         libxen-dev         uuid-dev         xfslibs-dev
>  ---> Running in a7f6ae892a05
> Reading package lists...
> Building dependency tree...
> Reading state information...
> E: Unable to locate package libxen-dev

Thanks for testing this, since the first steps passed, the following
shouldn't be a problem now.

Regards,

Phil.

Re: [Qemu-devel] [RFC PATCH 3/4] docker: Rename 'amd64' image as 'host-cross'

[Philippe Mathieu-Daudé]

On 07/17/2018 11:45 AM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .shippable.yml                                           | 4 ++--
>>  tests/docker/Makefile.include                            | 2 +-
>>  .../{debian-amd64.docker => debian-host-cross.docker}    | 9
>> ++++++---
> 
> I'd prefer debian-amd64->debian-host as it's not really a cross
> compiler.

This sounds better, indeed.

> Otherwise:
> 
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> 
> 
>>  3 files changed, 9 insertions(+), 6 deletions(-)
>>  rename tests/docker/dockerfiles/{debian-amd64.docker => debian-host-cross.docker} (82%)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index f74a3de3ff..7625ba8f27 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -5,8 +5,8 @@ env:
>>    global:
>>      - LC_ALL=C
>>    matrix:
>> -    - IMAGE=debian-amd64
>> -      TARGET_LIST=x86_64-softmmu,x86_64-linux-user
>> +    - IMAGE=debian-host-cross
>> +      TARGET_LIST=aarch64-softmmu,x86_64-softmmu,x86_64-linux-user

This line might be tuned a bit.

>>      - IMAGE=debian-win32-cross
>>        TARGET_LIST=arm-softmmu,i386-softmmu,lm32-softmmu
>>      - IMAGE=debian-win64-cross
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index b2a7e761cc..2db4190b4c 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -83,7 +83,7 @@ endif
>>  # Enforce dependencies for composite images
>>  docker-image-debian: docker-image-debian9
>>  docker-image-debian8-mxe: docker-image-debian8
>> -docker-image-debian-amd64: docker-image-debian9
>> +docker-image-debian-host-cross: docker-image-debian9
>>  docker-image-debian-armel-cross: docker-image-debian9
>>  docker-image-debian-armhf-cross: docker-image-debian9
>>  docker-image-debian-arm64-cross: docker-image-debian9
>> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-host-cross.docker
>> similarity index 82%
>> rename from tests/docker/dockerfiles/debian-amd64.docker
>> rename to tests/docker/dockerfiles/debian-host-cross.docker
>> index d79a599a0d..17d937efc9 100644
>> --- a/tests/docker/dockerfiles/debian-amd64.docker
>> +++ b/tests/docker/dockerfiles/debian-host-cross.docker
>> @@ -1,8 +1,11 @@
>>  #
>> -# Docker x86_64 target
>> +# Docker 'host' target (same architecture than the host)
>>  #
>> -# This docker target builds on the debian Stretch base image. Further
>> -# libraries which are not widely available are installed by hand.
>> +# This docker target builds on the debian Stretch base image.
>> +#
>> +# It is intented to build the host architecture, with as much
>> +# features as possible (libraries which are not widely available
>> +# are installed by hand).
>>  #
>>  FROM qemu:debian9
>>  MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> 
> --
> Alex Bennée
>