qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v1] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild"
@ 2021-01-25  9:03 Thomas Huth
  2021-01-25  9:51 ` Paolo Bonzini
  0 siblings, 1 reply; 6+ messages in thread
From: Thomas Huth @ 2021-01-25  9:03 UTC (permalink / raw)
  To: qemu-devel
  Cc: Paolo Bonzini, Daniel P . Berrangé,
	Wainer dos Santos Moschetta, Philippe Mathieu-Daudé

Currently, our check-system-* jobs are recompiling the whole sources
again. This happens due to the fact that the jobs are checking out
the whole source tree and required submodules again, and only try
to use the "build" directory with the binaries and object files
as an artifact from the previous stage - which simply does not work
anymore (with the current version of meson). Due to some changed
time stamps, meson is always trying to rebuild the whole tree.

To fix this problem, we can use "meson test --no-rebuild" instead of
make check" to avoid rebuilding all binaries every time. However, the
iotests ("make check-block") are not run by "meson test", so we have
to execute these manually now. But instead of adding them to the same
job as "meson test", it's better to put them into a separate new job
instead, to keep things nicely running in parallel in the CI.
This saves ca. 15 - 20 minutes of precious CI cycles in each run.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 RFC -> v1: Added iotests

 .gitlab-ci.yml | 113 ++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 94 insertions(+), 19 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index de3a3d25b5..0834267a37 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -34,6 +34,26 @@ include:
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
 
+.native_meson_test_job:
+  stage: test
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - cd build
+    - touch *
+    - make git-submodule-update
+    - if [ -x ../meson/meson.py ]; then
+          ../meson/meson.py test --no-rebuild -t 5 $MESON_TEST_ARGS ;
+      else
+          meson test --no-rebuild -t 5 $MESON_TEST_ARGS ;
+      fi
+
+.native_iotest_job:
+  stage: test
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - cd build/tests/qemu-iotests/
+    - ./check -g auto -qcow2
+
 .native_test_job_template: &native_test_job_definition
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
@@ -83,17 +103,15 @@ build-system-alpine:
   artifacts:
     expire_in: 2 days
     paths:
-      - .git-submodule-status
       - build
 
 check-system-alpine:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
   needs:
     - job: build-system-alpine
       artifacts: true
   variables:
     IMAGE: alpine
-    MAKE_CHECK_ARGS: check
 
 acceptance-system-alpine:
   <<: *native_test_job_definition
@@ -118,13 +136,20 @@ build-system-ubuntu:
       - build
 
 check-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
+  needs:
+    - job: build-system-ubuntu
+      artifacts: true
+  variables:
+    IMAGE: ubuntu2004
+
+iotest-ubuntu:
+  extends: .native_iotest_job
   needs:
     - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
-    MAKE_CHECK_ARGS: check
 
 acceptance-system-ubuntu:
   <<: *native_test_job_definition
@@ -149,13 +174,20 @@ build-system-debian:
       - build
 
 check-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
+  needs:
+    - job: build-system-debian
+      artifacts: true
+  variables:
+    IMAGE: debian-amd64
+
+iotest-debian:
+  extends: .native_iotest_job
   needs:
     - job: build-system-debian
       artifacts: true
   variables:
     IMAGE: debian-amd64
-    MAKE_CHECK_ARGS: check
 
 # No targets are built here, just tools, docs, and unit tests. This
 # also feeds into the eventual documentation deployment steps later
@@ -194,13 +226,20 @@ build-system-fedora:
       - build
 
 check-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
+  needs:
+    - job: build-system-fedora
+      artifacts: true
+  variables:
+    IMAGE: fedora
+
+iotest-fedora:
+  extends: .native_iotest_job
   needs:
     - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
-    MAKE_CHECK_ARGS: check
 
 acceptance-system-fedora:
   <<: *native_test_job_definition
@@ -226,13 +265,20 @@ build-system-centos:
       - build
 
 check-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos8
+
+iotest-centos:
+  extends: .native_iotest_job
   needs:
     - job: build-system-centos
       artifacts: true
   variables:
     IMAGE: centos8
-    MAKE_CHECK_ARGS: check
 
 acceptance-system-centos:
   <<: *native_test_job_definition
@@ -256,13 +302,20 @@ build-system-opensuse:
       - build
 
 check-system-opensuse:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
+  needs:
+    - job: build-system-opensuse
+      artifacts: true
+  variables:
+    IMAGE: opensuse-leap
+
+iotest-opensuse:
+  extends: .native_iotest_job
   needs:
     - job: build-system-opensuse
       artifacts: true
   variables:
     IMAGE: opensuse-leap
-    MAKE_CHECK_ARGS: check
 
 acceptance-system-opensuse:
    <<: *native_test_job_definition
@@ -525,13 +578,20 @@ build-crypto-old-nettle:
       - build
 
 check-crypto-old-nettle:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
+  needs:
+    - job: build-crypto-old-nettle
+      artifacts: true
+  variables:
+    IMAGE: centos7
+
+iotest-crypto-old-nettle:
+  extends: .native_iotest_job
   needs:
     - job: build-crypto-old-nettle
       artifacts: true
   variables:
     IMAGE: centos7
-    MAKE_CHECK_ARGS: check
 
 
 build-crypto-old-gcrypt:
@@ -546,13 +606,20 @@ build-crypto-old-gcrypt:
       - build
 
 check-crypto-old-gcrypt:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
+  needs:
+    - job: build-crypto-old-gcrypt
+      artifacts: true
+  variables:
+    IMAGE: centos7
+
+iotest-crypto-old-gcrypt:
+  extends: .native_iotest_job
   needs:
     - job: build-crypto-old-gcrypt
       artifacts: true
   variables:
     IMAGE: centos7
-    MAKE_CHECK_ARGS: check
 
 
 build-crypto-only-gnutls:
@@ -567,13 +634,21 @@ build-crypto-only-gnutls:
       - build
 
 check-crypto-only-gnutls:
-  <<: *native_test_job_definition
+  extends: .native_meson_test_job
   needs:
     - job: build-crypto-only-gnutls
       artifacts: true
   variables:
     IMAGE: centos7
-    MAKE_CHECK_ARGS: check
+
+iotest-crypto-only-gnutls:
+  extends: .native_iotest_job
+  needs:
+    - job: build-crypto-only-gnutls
+      artifacts: true
+  variables:
+    IMAGE: centos7
+
 
 # We don't need to exercise every backend with every front-end
 build-trace-multi-user:
-- 
2.27.0



^ permalink raw reply related	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-01-26  8:49 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-01-25  9:03 [PATCH v1] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild" Thomas Huth
2021-01-25  9:51 ` Paolo Bonzini
2021-01-25  9:56   ` Daniel P. Berrangé
2021-01-25 10:36     ` Paolo Bonzini
2021-01-26  7:03   ` Thomas Huth
2021-01-26  8:48     ` Paolo Bonzini

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).