[PATCH 0/4] gitlab-ci improvements

[PATCH 0/4] gitlab-ci improvements

[Thomas Huth]

Move the build-tools-and-docs-debian to a better place, add dependencies
and merge some jobs to speed up the CI.

Thomas Huth (4):
  gitlab-ci.yml: Move build-tools-and-docs-debian to a better place
  gitlab-ci.yml: Add some missing dependencies to the jobs
  gitlab-ci.yml: Merge one of the coroutine jobs with the tcg-disabled
    job
  gitlab-ci.yml: Merge check-crypto-old jobs into the build-crypto-old
    jobs

 .gitlab-ci.yml | 95 ++++++++++++++++----------------------------------
 1 file changed, 31 insertions(+), 64 deletions(-)

-- 
2.27.0

[PATCH 1/4] gitlab-ci.yml: Move build-tools-and-docs-debian to a better place

[Thomas Huth]

The "build-tools-and-docs-debian" job had been added in between
the "check-system-debian" and the "accepance-system-debian" jobs
and thus separates the jobs that belong together. Move it away,
to the end of the file, next to the "pages" job that depends on it.
And while we're at it, also add a proper "needs:" line to the
job so that it can be started as soon as possible instead of always
waiting for the previous stage to finish.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 07202f6ffb..871ea45a5f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -167,19 +167,6 @@ check-system-debian:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
-# No targets are built here, just tools, docs, and unit tests. This
-# also feeds into the eventual documentation deployment steps later
-build-tools-and-docs-debian:
-  <<: *native_build_job_definition
-  variables:
-    IMAGE: debian-amd64
-    MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
-    CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
 acceptance-system-debian:
   <<: *native_test_job_definition
   needs:
@@ -746,6 +733,21 @@ build-libvhost-user:
     - meson
     - ninja
 
+# No targets are built here, just tools, docs, and unit tests. This
+# also feeds into the eventual documentation deployment steps later
+build-tools-and-docs-debian:
+  <<: *native_build_job_definition
+  needs:
+    job: amd64-debian-container
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
+    CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
 # Prepare for GitLab pages deployment. Anything copied into the
 # "public" directory will be deployed to $USER.gitlab.io/$PROJECT
 pages:
-- 
2.27.0

[PATCH 2/4] gitlab-ci.yml: Add some missing dependencies to the jobs

[Thomas Huth]

Let's make sure that all jobs have proper "needs:" statements so that
they can start as soon as possible, without having to wait for the
previous pipeline stage to finish.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 871ea45a5f..a98f5674d6 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -432,6 +432,8 @@ build-user-plugins:
 
 build-user-centos7:
   <<: *native_build_job_definition
+  needs:
+    job: amd64-centos7-container
   variables:
     IMAGE: centos7
     CONFIGURE_ARGS: --disable-system --disable-tools --disable-docs
@@ -461,6 +463,8 @@ clang-system:
 
 clang-user:
   <<: *native_build_job_definition
+  needs:
+    job: amd64-debian-user-cross-container
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
@@ -470,6 +474,8 @@ clang-user:
 
 tsan-build:
   <<: *native_build_job_definition
+  needs:
+    job: amd64-ubuntu2004-container
   variables:
     IMAGE: ubuntu2004
     CONFIGURE_ARGS: --enable-tsan --cc=clang-10 --cxx=clang++-10 --disable-docs
@@ -507,6 +513,8 @@ check-deprecated:
 # gprof/gcov are GCC features
 gprof-gcov:
   <<: *native_build_job_definition
+  needs:
+    job: amd64-ubuntu2004-container
   variables:
     IMAGE: ubuntu2004
     CONFIGURE_ARGS: --enable-gprof --enable-gcov
-- 
2.27.0

[PATCH 3/4] gitlab-ci.yml: Merge one of the coroutine jobs with the tcg-disabled job

[Thomas Huth]

Our gitlab-ci got quite slow in the past weeks, due to the immense amount
of jobs that we have, so we should try to reduce the number of jobs.
Since we already have a job that builds without TCG, we can merge
one of the "build-coroutine" jobs with it to get rid of at least one
job.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a98f5674d6..45303cafdd 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -366,6 +366,8 @@ build-disabled:
 # Xen accelerator is not detected / selected. As result it build the
 # i386-softmmu and x86_64-softmmu with KVM being the single accelerator
 # available.
+# Also use a different coroutine implementation (which is only really of
+# interest to KVM users, i.e. with TCG disabled)
 build-tcg-disabled:
   <<: *native_build_job_definition
   needs:
@@ -375,7 +377,8 @@ build-tcg-disabled:
   script:
     - mkdir build
     - cd build
-    - ../configure --disable-tcg --audio-drv-list="" || { cat config.log meson-logs/meson-log.txt && exit 1; }
+    - ../configure --disable-tcg --audio-drv-list="" --with-coroutine=ucontext
+      || { cat config.log meson-logs/meson-log.txt && exit 1; }
     - make -j"$JOBS"
     - make check-unit
     - make check-qapi-schema
@@ -569,15 +572,6 @@ build-tci:
 
 # Alternate coroutines implementations are only really of interest to KVM users
 # However we can't test against KVM on Gitlab-CI so we can only run unit tests
-build-coroutine-ucontext:
-  <<: *native_build_job_definition
-  needs:
-    job: amd64-ubuntu2004-container
-  variables:
-    IMAGE: ubuntu2004
-    CONFIGURE_ARGS: --with-coroutine=ucontext --disable-tcg
-    MAKE_CHECK_ARGS: check-unit
-
 build-coroutine-sigaltstack:
   <<: *native_build_job_definition
   needs:
-- 
2.27.0

[PATCH 4/4] gitlab-ci.yml: Merge check-crypto-old jobs into the build-crypto-old jobs

[Thomas Huth]

Both, the build-crypto-old and the check-crypto-old jobs finish reasonably
fast, and the build artifacts are only used for the single corresponding
check jobs, so there is no reason for doing the check step in a separate
job here. Thus let's stop wasting artifacts space and job scheduler over-
head by simply merging the test step into the build jobs.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 45 ++++-----------------------------------------
 1 file changed, 4 insertions(+), 41 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 45303cafdd..0ada3dbb90 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -585,7 +585,7 @@ build-coroutine-sigaltstack:
 #
 # These jobs test old gcrypt and nettle from RHEL7
 # which had some API differences.
-build-crypto-old-nettle:
+crypto-old-nettle:
   <<: *native_build_job_definition
   needs:
     job: amd64-centos7-container
@@ -593,22 +593,9 @@ build-crypto-old-nettle:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
     CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    paths:
-      - build
-
-check-crypto-old-nettle:
-  <<: *native_test_job_definition
-  needs:
-    - job: build-crypto-old-nettle
-      artifacts: true
-  variables:
-    IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
-
-build-crypto-old-gcrypt:
+crypto-old-gcrypt:
   <<: *native_build_job_definition
   needs:
     job: amd64-centos7-container
@@ -616,22 +603,9 @@ build-crypto-old-gcrypt:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    paths:
-      - build
-
-check-crypto-old-gcrypt:
-  <<: *native_test_job_definition
-  needs:
-    - job: build-crypto-old-gcrypt
-      artifacts: true
-  variables:
-    IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
-
-build-crypto-only-gnutls:
+crypto-only-gnutls:
   <<: *native_build_job_definition
   needs:
     job: amd64-centos7-container
@@ -639,20 +613,9 @@ build-crypto-only-gnutls:
     IMAGE: centos7
     TARGETS: x86_64-softmmu x86_64-linux-user
     CONFIGURE_ARGS: --disable-nettle --disable-gcrypt --enable-gnutls
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    paths:
-      - build
-
-check-crypto-only-gnutls:
-  <<: *native_test_job_definition
-  needs:
-    - job: build-crypto-only-gnutls
-      artifacts: true
-  variables:
-    IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
+
 # We don't need to exercise every backend with every front-end
 build-trace-multi-user:
   <<: *native_build_job_definition
-- 
2.27.0

Re: [PATCH 0/4] gitlab-ci improvements

[Richard Henderson]

On 3/11/21 8:22 AM, Thomas Huth wrote:
> Thomas Huth (4):
>    gitlab-ci.yml: Move build-tools-and-docs-debian to a better place
>    gitlab-ci.yml: Add some missing dependencies to the jobs
>    gitlab-ci.yml: Merge one of the coroutine jobs with the tcg-disabled
>      job
>    gitlab-ci.yml: Merge check-crypto-old jobs into the build-crypto-old
>      jobs

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~