[PATCH] gitlab-ci.yml: Avoid some submodules to speed up the CI a little bit

[PATCH] gitlab-ci.yml: Avoid some submodules to speed up the CI a little bit

[Thomas Huth]

Since the meson build system rework, the configure script prefers the
git submodules over the system libraries. So we are testing compilation
with capstone, fdt and libslirp as a submodule all over the place,
burning CPU cycles by recompiling these third party modules and wasting
some network bandwidth in the CI by cloning the submodules each time.
Let's stop doing that in at least a couple of jobs and use the system
libraries instead.

While we're at it, also install meson in the Fedora container, since
it is new enough already, so we do not need to check out the meson
submodule here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml                               | 6 +++++-
 tests/docker/dockerfiles/centos8.docker      | 1 +
 tests/docker/dockerfiles/debian-amd64.docker | 1 +
 tests/docker/dockerfiles/fedora.docker       | 3 +++
 tests/docker/dockerfiles/ubuntu2004.docker   | 1 +
 5 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index de3a3d25b5..e2f9c99e27 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -109,6 +109,7 @@ build-system-ubuntu:
   <<: *native_build_job_definition
   variables:
     IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --enable-fdt=system --enable-slirp=system
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
       moxie-softmmu microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -140,6 +141,7 @@ build-system-debian:
   <<: *native_build_job_definition
   variables:
     IMAGE: debian-amd64
+    CONFIGURE_ARGS: --enable-fdt=system
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
       riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -185,6 +187,7 @@ build-system-fedora:
   variables:
     IMAGE: fedora
     CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
+             --enable-fdt=system --enable-slirp=system --enable-capstone=system
     TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu
       xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -216,7 +219,7 @@ build-system-centos:
   <<: *native_build_job_definition
   variables:
     IMAGE: centos8
-    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
+    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
     TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
@@ -248,6 +251,7 @@ build-system-opensuse:
   <<: *native_build_job_definition
   variables:
     IMAGE: opensuse-leap
+    CONFIGURE_ARGS: --enable-fdt=system
     TARGETS: s390x-softmmu x86_64-softmmu aarch64-softmmu
     MAKE_CHECK_ARGS: check-build
   artifacts:
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index 64cb7a6eda..a763d55730 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -15,6 +15,7 @@ ENV PACKAGES \
     glib2-devel \
     libaio-devel \
     libepoxy-devel \
+    libfdt-devel \
     libgcrypt-devel \
     lzo-devel \
     make \
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index a98314757d..ed546edcd6 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -21,6 +21,7 @@ RUN apt update && \
         libbz2-dev \
         liblzo2-dev \
         libgcrypt20-dev \
+        libfdt-dev \
         librdmacm-dev \
         libsasl2-dev \
         libsnappy-dev \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 0b5053f2d0..0d7602abbe 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -6,6 +6,7 @@ ENV PACKAGES \
     brlapi-devel \
     bzip2 \
     bzip2-devel \
+    capstone-devel \
     ccache \
     clang \
     cyrus-sasl-devel \
@@ -37,6 +38,7 @@ ENV PACKAGES \
     libpng-devel \
     librbd-devel \
     libseccomp-devel \
+    libslirp-devel \
     libssh-devel \
     libubsan \
     libudev-devel \
@@ -46,6 +48,7 @@ ENV PACKAGES \
     llvm \
     lzo-devel \
     make \
+    meson \
     mingw32-bzip2 \
     mingw32-curl \
     mingw32-glib2 \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index ae889d8482..8519584d2b 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -37,6 +37,7 @@ ENV PACKAGES flex bison \
     libsasl2-dev \
     libsdl2-dev \
     libseccomp-dev \
+    libslirp-dev \
     libsnappy-dev \
     libspice-protocol-dev \
     libspice-server-dev \
-- 
2.27.0

Re: [PATCH] gitlab-ci.yml: Avoid some submodules to speed up the CI a little bit

[Philippe Mathieu-Daudé]

Hi Thomas,

On 1/21/21 6:44 PM, Thomas Huth wrote:
> Since the meson build system rework, the configure script prefers the
> git submodules over the system libraries. So we are testing compilation
> with capstone, fdt and libslirp as a submodule all over the place,
> burning CPU cycles by recompiling these third party modules and wasting
> some network bandwidth in the CI by cloning the submodules each time.
> Let's stop doing that in at least a couple of jobs and use the system
> libraries instead.
> 
> While we're at it, also install meson in the Fedora container, since
> it is new enough already, so we do not need to check out the meson
> submodule here.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.yml                               | 6 +++++-
>  tests/docker/dockerfiles/centos8.docker      | 1 +
>  tests/docker/dockerfiles/debian-amd64.docker | 1 +
>  tests/docker/dockerfiles/fedora.docker       | 3 +++
>  tests/docker/dockerfiles/ubuntu2004.docker   | 1 +
>  5 files changed, 11 insertions(+), 1 deletion(-)

So now which are the jobs left building from git submodule clone?

Re: [PATCH] gitlab-ci.yml: Avoid some submodules to speed up the CI a little bit

[Thomas Huth]

On 26/01/2021 13.08, Philippe Mathieu-Daudé wrote:
> Hi Thomas,
> 
> On 1/21/21 6:44 PM, Thomas Huth wrote:
>> Since the meson build system rework, the configure script prefers the
>> git submodules over the system libraries. So we are testing compilation
>> with capstone, fdt and libslirp as a submodule all over the place,
>> burning CPU cycles by recompiling these third party modules and wasting
>> some network bandwidth in the CI by cloning the submodules each time.
>> Let's stop doing that in at least a couple of jobs and use the system
>> libraries instead.
>>
>> While we're at it, also install meson in the Fedora container, since
>> it is new enough already, so we do not need to check out the meson
>> submodule here.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   .gitlab-ci.yml                               | 6 +++++-
>>   tests/docker/dockerfiles/centos8.docker      | 1 +
>>   tests/docker/dockerfiles/debian-amd64.docker | 1 +
>>   tests/docker/dockerfiles/fedora.docker       | 3 +++
>>   tests/docker/dockerfiles/ubuntu2004.docker   | 1 +
>>   5 files changed, 11 insertions(+), 1 deletion(-)
> 
> So now which are the jobs left building from git submodule clone?

All the jobs which don't have a depending test stage jobs should still use 
the submodules, e.g. build-oss-fuzz, build-tci, etc.
Also the centos7-based jobs still use all submodules, e.g. 
build-crypto-old-nettle etc.

  Thomas