[PATCH 0/7] RFC: gitlab: enable use of ccache in FreeBSD / macOS Cirrus CI jobs

[PATCH 0/7] RFC: gitlab: enable use of ccache in FreeBSD / macOS Cirrus CI jobs

[Daniel P. Berrangé]

We've seen that we exhaust the Cirrus CI free credits for the QEMU
account, especially during the end of a release cycle.

It is hard to cut down the CI credits burn beyond its current level
without dropping features from the build configuration, which harms
CI coverage.

This proposes two things

 * First, when CI credits are exhausted we stop treating Cirrus CI
   failures as fatal. Instead the FreeBSD/macOS jobs get marked as
   "warning" meaning they won't break the whole pipeline.

 * Enable the use of ccache in the build process which would reduce
   the wallclock time in jobs, when there is a good cache hit rate.

...alot of heavy lifting is done by the last part of that sentence
above, meaning this isn't a guaranteed win. When ccache takes a
cache miss, compile time is worse than without ccache. Also we have
a time penalty of uploading/downloading the cache at start/end of
the job.

Observed timings over a number of runs:

 * No ccache (ie today)

    *   macOS: 6m00 build,  8m00 total
    *   macOS: 6m30 build,  8m20 total
    * FreeBSD: 8m45 build, 14m50 total
    * FreeBSD: 9m00 build, 15m30 total

   In macOS we see approx 2m for brew install tasks
   In FreeBSD we see approx 5m30-6m for pkg install tasks

 * ccache: cold cache (ie 0% hit rate)

    *   macOS: 8m15 build, 12m00 total
    *   macOS: 7m55 build, 10m03 total
    * FreeBSD: 9m35 build, 16m00 total
    * FreeBSD: 9m23 build, 15m15 total

   In macOS we see the compile time increase by 1m30-2m
   In FreeBSD we see the compile time increase 0m30-1m

 * ccache: hot cache (ie 99% hit rate)

    *   macOS: 5m25 build, 8m37 total
    *   macOS: 3m28 build, 5m35 total
    *   macOS: 3m40 build, 6m00 total
    * FreeBSD: 5m31 build, 12m00 total
    * FreeBSD: 5m25 build, 11m50 total
    * FreeBSD: 5m20 build, 11m00 total

   In macOS we see compile time decrease 0m30-3m
   In FreeBSD we see compile time decrease 3m

   Most of the "build" time remaining here is actually
   running of unit tests & qtests.

   There is some non-deterministic upload/download overhead
   between 0m05-0m30.

What this tells us:

  * ccache with cold cache is always slower than today
  * ccache with hot cache is usually better than today

The $1 million question is what our cache hit rate will look like
across real world pull requests ?

Also any slowness in homebrew / ports can impact our pkg install
times non-deterministically.

It is very hard to say for sure if this is a win, without actually
trying it over a release cycle and seeing how our CI credits usage
varies over 4 months. Even that will be hard, as we QEMU is a
moving target adding features. eg I wonder what Rust would do to
build times, if anything measurable ?

So our options

 * Trial ccache with Cirrus CI
 * Cull more features from macOS/FreeBSD build config
 * Reduce 'make check' coverage for macOS/FreeBSD
 * Get our own CI private runners for FreeBSD and use Cirrus
   only for macOS
 * <Your idea here>

Daniel P. Berrangé (7):
  gitlab: don't fail cirrus CI jobs when credits are exhausted
  gitlab: use new(ish) cirrus-vars command for creating config
  gitlab: clean packages in cirrus tasks
  gitlab: purge build files from cirrus CI jobs
  gitlab: enable ccache with Cirrus CI
  tests: update libvirt-ci submodule for newer ccache
  gitlab: force ccache to validate compiler version

 .gitlab-ci.d/cirrus.yml             | 30 +++++++++--------------------
 .gitlab-ci.d/cirrus/build.yml       | 14 +++++++++++++-
 .gitlab-ci.d/cirrus/freebsd-14.vars |  2 +-
 tests/lcitool/libvirt-ci            |  2 +-
 tests/vm/generated/freebsd.json     |  2 +-
 5 files changed, 25 insertions(+), 25 deletions(-)

-- 
2.46.0

[PATCH 1/7] gitlab: don't fail cirrus CI jobs when credits are exhausted

[Daniel P. Berrangé]

In the last week of the month we have often run out of credits on
Cirrus CI, which causes the jobs to fail, in turn causing the
overall pipeline to fail.

The cirrus-run tool can now detect the "out of credits" scenario
and exits with a code of '3'.  We can tell gitlab to treat this
exit code as special and mark the job as "warning" instead of
"failed". This allows the pipeline status overall to remain
green, when we have non-technical issues with Cirrus CI.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index a9e43e21d0..adc0007e5d 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -15,6 +15,8 @@
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:latest
   needs: []
+  allow_failure:
+    exit_codes: 3
   # 20 mins larger than "timeout_in" in cirrus/build.yml
   # as there's often a 5-10 minute delay before Cirrus CI
   # actually starts the task
-- 
2.46.0

[PATCH 2/7] gitlab: use new(ish) cirrus-vars command for creating config

[Daniel P. Berrangé]

Rather than a giant sed command with a hardcoded list of env var name,
we can now use the new(ish) cirrus-vars command that libvirt has added
to the 'cirrus-run' container.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus.yml       | 23 +++--------------------
 .gitlab-ci.d/cirrus/build.yml |  2 +-
 2 files changed, 4 insertions(+), 21 deletions(-)

diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index adc0007e5d..16411f3d2b 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -22,27 +22,10 @@
   # actually starts the task
   timeout: 80m
   script:
+    - set -o allexport
     - source .gitlab-ci.d/cirrus/$NAME.vars
-    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
-          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
-          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
-          -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
-          -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
-          -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
-          -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
-          -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
-          -e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
-          -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
-          -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g"
-          -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g"
-          -e "s|[@]PKGS@|$PKGS|g"
-          -e "s|[@]MAKE@|$MAKE|g"
-          -e "s|[@]PYTHON@|$PYTHON|g"
-          -e "s|[@]PIP3@|$PIP3|g"
-          -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g"
-          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
-          -e "s|[@]TEST_TARGETS@|$TEST_TARGETS|g"
-      <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
+    - set +o allexport
+    - cirrus-vars <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
   variables:
diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index 102cdbd8b1..41abd0b31a 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -8,7 +8,7 @@ env:
   CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
   CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
   CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
-  PATH: "@PATH@"
+  PATH: "@PATH_EXTRA@:$PATH"
   PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@"
   PYTHON: "@PYTHON@"
   MAKE: "@MAKE@"
-- 
2.46.0

[PATCH 3/7] gitlab: clean packages in cirrus tasks

[Daniel P. Berrangé]

The FreeBSD VM is somewhat low on disk space after all QEMU build deps
are installed and a full QEMU build performed. Purging the package
manager cache is a simple thing that reclaims about 1 GB of space.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus.yml       | 2 ++
 .gitlab-ci.d/cirrus/build.yml | 1 +
 2 files changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 16411f3d2b..2bd3cb35c9 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -42,6 +42,7 @@ x64-freebsd-14-build:
     CIRRUS_VM_RAM: 8G
     UPDATE_COMMAND: pkg update; pkg upgrade -y
     INSTALL_COMMAND: pkg install -y
+    CLEAN_COMMAND: pkg clean -y --all
     CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4eb-softmmu,xtensa-softmmu
     TEST_TARGETS: check
 
@@ -54,6 +55,7 @@ aarch64-macos-build:
     CIRRUS_VM_IMAGE_NAME: ghcr.io/cirruslabs/macos-runner:sonoma
     UPDATE_COMMAND: brew update
     INSTALL_COMMAND: brew install
+    CLEAN_COMMAND: brew cleanup --prune=all
     PATH_EXTRA: /opt/homebrew/ccache/libexec:/opt/homebrew/gettext/bin
     PKG_CONFIG_PATH: /opt/homebrew/curl/lib/pkgconfig:/opt/homebrew/ncurses/lib/pkgconfig:/opt/homebrew/readline/lib/pkgconfig
     CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblazeel-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4-softmmu,xtensaeb-softmmu
diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index 41abd0b31a..9983ab0690 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -21,6 +21,7 @@ build_task:
   install_script:
     - @UPDATE_COMMAND@
     - @INSTALL_COMMAND@ @PKGS@
+    - @CLEAN_COMMAND@
     - if test -n "@PYPI_PKGS@" ; then PYLIB=$(@PYTHON@ -c 'import sysconfig; print(sysconfig.get_path("stdlib"))'); rm -f $PYLIB/EXTERNALLY-MANAGED; @PIP3@ install @PYPI_PKGS@ ; fi
   clone_script:
     - git clone --depth 100 "$CI_REPOSITORY_URL" .
-- 
2.46.0

[PATCH 4/7] gitlab: purge build files from cirrus CI jobs

[Daniel P. Berrangé]

Uploading artifacts in Cirrus CI requires sufficient disk space to
create a tarball of the artifact files. IOW, whatever size the
artifacts are, double that. This results in space pressure on the
FreeBSD jobs due to limited disk size. Purging the .o files from
the meson build directory reclaims significant space.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus/build.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index 9983ab0690..d26a2a788c 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -37,6 +37,7 @@ build_task:
       do
         $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
       done
+    - find . -not -path 'meson-logs/*' -delete
   always:
     build_result_artifacts:
       path: build/meson-logs/*log.txt
-- 
2.46.0

[PATCH 5/7] gitlab: enable ccache with Cirrus CI

[Daniel P. Berrangé]

Add rules to configure the use of ccache with Cirrus CI jobs,
and preserve the cache across jobs.

The HomeBrew PATH was already present, but incorrect, while
the FreeBSD PATH was missing.

About 1 GB is enough to get a 99% hit rate on a pristine rebuild
with no code changes. Setting it much higher than this will
trigger ENOSPC problems on the FreeBSD builders due to limited
disk size.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus.yml       | 3 ++-
 .gitlab-ci.d/cirrus/build.yml | 9 +++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 2bd3cb35c9..af20082a01 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -43,6 +43,7 @@ x64-freebsd-14-build:
     UPDATE_COMMAND: pkg update; pkg upgrade -y
     INSTALL_COMMAND: pkg install -y
     CLEAN_COMMAND: pkg clean -y --all
+    PATH_EXTRA: /usr/local/libexec/ccache
     CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4eb-softmmu,xtensa-softmmu
     TEST_TARGETS: check
 
@@ -56,7 +57,7 @@ aarch64-macos-build:
     UPDATE_COMMAND: brew update
     INSTALL_COMMAND: brew install
     CLEAN_COMMAND: brew cleanup --prune=all
-    PATH_EXTRA: /opt/homebrew/ccache/libexec:/opt/homebrew/gettext/bin
+    PATH_EXTRA: /opt/homebrew/opt/ccache/libexec:/opt/homebrew/gettext/bin
     PKG_CONFIG_PATH: /opt/homebrew/curl/lib/pkgconfig:/opt/homebrew/ncurses/lib/pkgconfig:/opt/homebrew/readline/lib/pkgconfig
     CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblazeel-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4-softmmu,xtensaeb-softmmu
     TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index d26a2a788c..5c86278bf8 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -5,6 +5,7 @@
 
 env:
   CIRRUS_CLONE_DEPTH: 1
+  CCACHE: "@CCACHE@"
   CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
   CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
   CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
@@ -27,7 +28,14 @@ build_task:
     - git clone --depth 100 "$CI_REPOSITORY_URL" .
     - git fetch origin "$CI_COMMIT_REF_NAME"
     - git reset --hard "$CI_COMMIT_SHA"
+  obj_cache:
+    folder: ccache
   step_script:
+    - export CCACHE_BASEDIR="$(pwd)"
+    - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
+    - export CCACHE_MAXSIZE="1000M"
+    - ccache --zero-stats
+    - ccache --show-stats
     - mkdir build
     - cd build
     - ../configure --enable-werror $CONFIGURE_ARGS
@@ -38,6 +46,7 @@ build_task:
         $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
       done
     - find . -not -path 'meson-logs/*' -delete
+    - ccache --show-stats
   always:
     build_result_artifacts:
       path: build/meson-logs/*log.txt
-- 
2.46.0

[PATCH 6/7] tests: update libvirt-ci submodule for newer ccache

[Daniel P. Berrangé]

This update pulls in ccache4 for FreeBSD, to match other platforms

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus/freebsd-14.vars | 2 +-
 tests/lcitool/libvirt-ci            | 2 +-
 tests/vm/generated/freebsd.json     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/cirrus/freebsd-14.vars b/.gitlab-ci.d/cirrus/freebsd-14.vars
index 0a7ac5e0e1..0997c47af5 100644
--- a/.gitlab-ci.d/cirrus/freebsd-14.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-14.vars
@@ -11,6 +11,6 @@ MAKE='/usr/local/bin/gmake'
 NINJA='/usr/local/bin/ninja'
 PACKAGING_COMMAND='pkg'
 PIP3='/usr/local/bin/pip'
-PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk-vnc gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson mtools ncurses nettle ninja opencv pixman pkgconf png py311-numpy py311-pillow py311-pip py311-pyyaml py311-sphinx py311-sphinx_rtd_theme py311-tomli python3 rpm2cpio rust rust-bindgen-cli sdl2 sdl2_image snappy sndio socat spice-protocol tesseract usbredir virglrenderer vte3 xorriso zstd'
+PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache4 cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk-vnc gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson mtools ncurses nettle ninja opencv pixman pkgconf png py311-numpy py311-pillow py311-pip py311-pyyaml py311-sphinx py311-sphinx_rtd_theme py311-tomli python3 rpm2cpio rust rust-bindgen-cli sdl2 sdl2_image snappy sndio socat spice-protocol tesseract usbredir virglrenderer vte3 xorriso zstd'
 PYPI_PKGS=''
 PYTHON='/usr/local/bin/python3'
diff --git a/tests/lcitool/libvirt-ci b/tests/lcitool/libvirt-ci
index 9ad3f70bde..0f11966131 160000
--- a/tests/lcitool/libvirt-ci
+++ b/tests/lcitool/libvirt-ci
@@ -1 +1 @@
-Subproject commit 9ad3f70bde9865d5ad18f36d256d472e72b5cbf3
+Subproject commit 0f119661317333038e91b6fb9d0381a6934dcd0c
diff --git a/tests/vm/generated/freebsd.json b/tests/vm/generated/freebsd.json
index 3cb7fb7060..81fc38d798 100644
--- a/tests/vm/generated/freebsd.json
+++ b/tests/vm/generated/freebsd.json
@@ -13,7 +13,7 @@
     "bzip2",
     "ca_root_nss",
     "capstone4",
-    "ccache",
+    "ccache4",
     "cmocka",
     "ctags",
     "curl",
-- 
2.46.0

[PATCH 7/7] gitlab: force ccache to validate compiler version

[Daniel P. Berrangé]

By default ccache checks the compiler 'mtime' to determine if it should
invalidate the cache. On FreeBSD the 'mtime' reflects when the compiler
package was installed, rather than when it was built. IOW, on throwaway
CI VMs, the 'mtime' changes on every single job and is thus useless.

It could validate the compiler binary content, but validating the
compiler version string is less CPU intensive.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus/build.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index 5c86278bf8..8268e9e547 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -34,6 +34,7 @@ build_task:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
     - export CCACHE_MAXSIZE="1000M"
+    - export CCACHE_COMPILERCHECK="string:$(clang -v 2>&1)"
     - ccache --zero-stats
     - ccache --show-stats
     - mkdir build
-- 
2.46.0

Re: [PATCH 4/7] gitlab: purge build files from cirrus CI jobs

[Philippe Mathieu-Daudé]

On 4/12/24 20:48, Daniel P. Berrangé wrote:
> Uploading artifacts in Cirrus CI requires sufficient disk space to
> create a tarball of the artifact files. IOW, whatever size the
> artifacts are, double that. This results in space pressure on the
> FreeBSD jobs due to limited disk size. Purging the .o files from
> the meson build directory reclaims significant space.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus/build.yml | 1 +
>   1 file changed, 1 insertion(+)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 7/7] gitlab: force ccache to validate compiler version

[Philippe Mathieu-Daudé]

On 4/12/24 20:48, Daniel P. Berrangé wrote:
> By default ccache checks the compiler 'mtime' to determine if it should
> invalidate the cache. On FreeBSD the 'mtime' reflects when the compiler
> package was installed, rather than when it was built. IOW, on throwaway
> CI VMs, the 'mtime' changes on every single job and is thus useless.
> 
> It could validate the compiler binary content, but validating the
> compiler version string is less CPU intensive.
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus/build.yml | 1 +
>   1 file changed, 1 insertion(+)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 1/7] gitlab: don't fail cirrus CI jobs when credits are exhausted

[Thomas Huth]

On 04/12/2024 20.48, Daniel P. Berrangé wrote:
> In the last week of the month we have often run out of credits on
> Cirrus CI, which causes the jobs to fail, in turn causing the
> overall pipeline to fail.
> 
> The cirrus-run tool can now detect the "out of credits" scenario
> and exits with a code of '3'.  We can tell gitlab to treat this
> exit code as special and mark the job as "warning" instead of
> "failed". This allows the pipeline status overall to remain
> green, when we have non-technical issues with Cirrus CI.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus.yml | 2 ++
>   1 file changed, 2 insertions(+)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 3/7] gitlab: clean packages in cirrus tasks

[Thomas Huth]

On 04/12/2024 20.48, Daniel P. Berrangé wrote:
> The FreeBSD VM is somewhat low on disk space after all QEMU build deps
> are installed and a full QEMU build performed. Purging the package
> manager cache is a simple thing that reclaims about 1 GB of space.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus.yml       | 2 ++
>   .gitlab-ci.d/cirrus/build.yml | 1 +
>   2 files changed, 3 insertions(+)
> 
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> index 16411f3d2b..2bd3cb35c9 100644
> --- a/.gitlab-ci.d/cirrus.yml
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -42,6 +42,7 @@ x64-freebsd-14-build:
>       CIRRUS_VM_RAM: 8G
>       UPDATE_COMMAND: pkg update; pkg upgrade -y
>       INSTALL_COMMAND: pkg install -y
> +    CLEAN_COMMAND: pkg clean -y --all
>       CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4eb-softmmu,xtensa-softmmu
>       TEST_TARGETS: check
>   
> @@ -54,6 +55,7 @@ aarch64-macos-build:
>       CIRRUS_VM_IMAGE_NAME: ghcr.io/cirruslabs/macos-runner:sonoma
>       UPDATE_COMMAND: brew update
>       INSTALL_COMMAND: brew install
> +    CLEAN_COMMAND: brew cleanup --prune=all

Are we also short on disk space in the macOS jobs? Otherwise, I wonder 
whether we should rather skip the step here to save some seconds of run time?

  Thomas


>       PATH_EXTRA: /opt/homebrew/ccache/libexec:/opt/homebrew/gettext/bin
>       PKG_CONFIG_PATH: /opt/homebrew/curl/lib/pkgconfig:/opt/homebrew/ncurses/lib/pkgconfig:/opt/homebrew/readline/lib/pkgconfig
>       CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblazeel-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4-softmmu,xtensaeb-softmmu
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> index 41abd0b31a..9983ab0690 100644
> --- a/.gitlab-ci.d/cirrus/build.yml
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -21,6 +21,7 @@ build_task:
>     install_script:
>       - @UPDATE_COMMAND@
>       - @INSTALL_COMMAND@ @PKGS@
> +    - @CLEAN_COMMAND@
>       - if test -n "@PYPI_PKGS@" ; then PYLIB=$(@PYTHON@ -c 'import sysconfig; print(sysconfig.get_path("stdlib"))'); rm -f $PYLIB/EXTERNALLY-MANAGED; @PIP3@ install @PYPI_PKGS@ ; fi
>     clone_script:
>       - git clone --depth 100 "$CI_REPOSITORY_URL" .

Re: [PATCH 3/7] gitlab: clean packages in cirrus tasks

[Daniel P. Berrangé]

On Thu, Dec 05, 2024 at 09:58:54AM +0100, Thomas Huth wrote:
> On 04/12/2024 20.48, Daniel P. Berrangé wrote:
> > The FreeBSD VM is somewhat low on disk space after all QEMU build deps
> > are installed and a full QEMU build performed. Purging the package
> > manager cache is a simple thing that reclaims about 1 GB of space.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/cirrus.yml       | 2 ++
> >   .gitlab-ci.d/cirrus/build.yml | 1 +
> >   2 files changed, 3 insertions(+)
> > 
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > index 16411f3d2b..2bd3cb35c9 100644
> > --- a/.gitlab-ci.d/cirrus.yml
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -42,6 +42,7 @@ x64-freebsd-14-build:
> >       CIRRUS_VM_RAM: 8G
> >       UPDATE_COMMAND: pkg update; pkg upgrade -y
> >       INSTALL_COMMAND: pkg install -y
> > +    CLEAN_COMMAND: pkg clean -y --all
> >       CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4eb-softmmu,xtensa-softmmu
> >       TEST_TARGETS: check
> > @@ -54,6 +55,7 @@ aarch64-macos-build:
> >       CIRRUS_VM_IMAGE_NAME: ghcr.io/cirruslabs/macos-runner:sonoma
> >       UPDATE_COMMAND: brew update
> >       INSTALL_COMMAND: brew install
> > +    CLEAN_COMMAND: brew cleanup --prune=all
> 
> Are we also short on disk space in the macOS jobs? Otherwise, I wonder
> whether we should rather skip the step here to save some seconds of run
> time?

I've not measured it, but I've not seen disk space issues on macOS. Still
this command is quick and lost in the noise of the package install process
which will vary depending on network performance and homebrew server load.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 3/7] gitlab: clean packages in cirrus tasks

[Thomas Huth]

On 05/12/2024 10.19, Daniel P. Berrangé wrote:
> On Thu, Dec 05, 2024 at 09:58:54AM +0100, Thomas Huth wrote:
>> On 04/12/2024 20.48, Daniel P. Berrangé wrote:
>>> The FreeBSD VM is somewhat low on disk space after all QEMU build deps
>>> are installed and a full QEMU build performed. Purging the package
>>> manager cache is a simple thing that reclaims about 1 GB of space.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>    .gitlab-ci.d/cirrus.yml       | 2 ++
>>>    .gitlab-ci.d/cirrus/build.yml | 1 +
>>>    2 files changed, 3 insertions(+)
>>>
>>> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
>>> index 16411f3d2b..2bd3cb35c9 100644
>>> --- a/.gitlab-ci.d/cirrus.yml
>>> +++ b/.gitlab-ci.d/cirrus.yml
>>> @@ -42,6 +42,7 @@ x64-freebsd-14-build:
>>>        CIRRUS_VM_RAM: 8G
>>>        UPDATE_COMMAND: pkg update; pkg upgrade -y
>>>        INSTALL_COMMAND: pkg install -y
>>> +    CLEAN_COMMAND: pkg clean -y --all
>>>        CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4eb-softmmu,xtensa-softmmu
>>>        TEST_TARGETS: check
>>> @@ -54,6 +55,7 @@ aarch64-macos-build:
>>>        CIRRUS_VM_IMAGE_NAME: ghcr.io/cirruslabs/macos-runner:sonoma
>>>        UPDATE_COMMAND: brew update
>>>        INSTALL_COMMAND: brew install
>>> +    CLEAN_COMMAND: brew cleanup --prune=all
>>
>> Are we also short on disk space in the macOS jobs? Otherwise, I wonder
>> whether we should rather skip the step here to save some seconds of run
>> time?
> 
> I've not measured it, but I've not seen disk space issues on macOS. Still
> this command is quick and lost in the noise of the package install process
> which will vary depending on network performance and homebrew server load.

Ok, fair, so if it is a quick command:

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 4/7] gitlab: purge build files from cirrus CI jobs

[Thomas Huth]

On 04/12/2024 20.48, Daniel P. Berrangé wrote:
> Uploading artifacts in Cirrus CI requires sufficient disk space to
> create a tarball of the artifact files. IOW, whatever size the
> artifacts are, double that. This results in space pressure on the
> FreeBSD jobs due to limited disk size. Purging the .o files from
> the meson build directory reclaims significant space.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus/build.yml | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> index 9983ab0690..d26a2a788c 100644
> --- a/.gitlab-ci.d/cirrus/build.yml
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -37,6 +37,7 @@ build_task:
>         do
>           $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
>         done
> +    - find . -not -path 'meson-logs/*' -delete

I'm not sure, but this might cause trouble if you run the Cirrus-CI job with 
terminal access for testing the binaries manually after the build succeeded? 
Maybe it would be better to just kill the .o files and leave the rest around?

  Thomas

Re: [PATCH 5/7] gitlab: enable ccache with Cirrus CI

[Thomas Huth]

On 04/12/2024 20.48, Daniel P. Berrangé wrote:
> Add rules to configure the use of ccache with Cirrus CI jobs,
> and preserve the cache across jobs.
> 
> The HomeBrew PATH was already present, but incorrect, while
> the FreeBSD PATH was missing.
> 
> About 1 GB is enough to get a 99% hit rate on a pristine rebuild
> with no code changes. Setting it much higher than this will
> trigger ENOSPC problems on the FreeBSD builders due to limited
> disk size.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus.yml       | 3 ++-
>   .gitlab-ci.d/cirrus/build.yml | 9 +++++++++
>   2 files changed, 11 insertions(+), 1 deletion(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 3/7] gitlab: clean packages in cirrus tasks

[Daniel P. Berrangé]

On Thu, Dec 05, 2024 at 11:09:01AM +0100, Thomas Huth wrote:
> On 05/12/2024 10.19, Daniel P. Berrangé wrote:
> > On Thu, Dec 05, 2024 at 09:58:54AM +0100, Thomas Huth wrote:
> > > On 04/12/2024 20.48, Daniel P. Berrangé wrote:
> > > > The FreeBSD VM is somewhat low on disk space after all QEMU build deps
> > > > are installed and a full QEMU build performed. Purging the package
> > > > manager cache is a simple thing that reclaims about 1 GB of space.
> > > > 
> > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > > ---
> > > >    .gitlab-ci.d/cirrus.yml       | 2 ++
> > > >    .gitlab-ci.d/cirrus/build.yml | 1 +
> > > >    2 files changed, 3 insertions(+)
> > > > 
> > > > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > > > index 16411f3d2b..2bd3cb35c9 100644
> > > > --- a/.gitlab-ci.d/cirrus.yml
> > > > +++ b/.gitlab-ci.d/cirrus.yml
> > > > @@ -42,6 +42,7 @@ x64-freebsd-14-build:
> > > >        CIRRUS_VM_RAM: 8G
> > > >        UPDATE_COMMAND: pkg update; pkg upgrade -y
> > > >        INSTALL_COMMAND: pkg install -y
> > > > +    CLEAN_COMMAND: pkg clean -y --all
> > > >        CONFIGURE_ARGS: --target-list-exclude=arm-softmmu,i386-softmmu,microblaze-softmmu,mips64el-softmmu,mipsel-softmmu,mips-softmmu,ppc-softmmu,sh4eb-softmmu,xtensa-softmmu
> > > >        TEST_TARGETS: check
> > > > @@ -54,6 +55,7 @@ aarch64-macos-build:
> > > >        CIRRUS_VM_IMAGE_NAME: ghcr.io/cirruslabs/macos-runner:sonoma
> > > >        UPDATE_COMMAND: brew update
> > > >        INSTALL_COMMAND: brew install
> > > > +    CLEAN_COMMAND: brew cleanup --prune=all
> > > 
> > > Are we also short on disk space in the macOS jobs? Otherwise, I wonder
> > > whether we should rather skip the step here to save some seconds of run
> > > time?
> > 
> > I've not measured it, but I've not seen disk space issues on macOS. Still
> > this command is quick and lost in the noise of the package install process
> > which will vary depending on network performance and homebrew server load.
> 
> Ok, fair, so if it is a quick command:

I measured it now.  6 seconds run time, frees 2 GB of space on macOS.

The macOS VM has ~300 GB working space just for builds, which is an order
of magnitude more than the FreeBSD VMs get at 20 GB for the entire OS
install and builds.

> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 4/7] gitlab: purge build files from cirrus CI jobs

[Daniel P. Berrangé]

On Thu, Dec 05, 2024 at 11:55:00AM +0100, Thomas Huth wrote:
> On 04/12/2024 20.48, Daniel P. Berrangé wrote:
> > Uploading artifacts in Cirrus CI requires sufficient disk space to
> > create a tarball of the artifact files. IOW, whatever size the
> > artifacts are, double that. This results in space pressure on the
> > FreeBSD jobs due to limited disk size. Purging the .o files from
> > the meson build directory reclaims significant space.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/cirrus/build.yml | 1 +
> >   1 file changed, 1 insertion(+)
> > 
> > diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> > index 9983ab0690..d26a2a788c 100644
> > --- a/.gitlab-ci.d/cirrus/build.yml
> > +++ b/.gitlab-ci.d/cirrus/build.yml
> > @@ -37,6 +37,7 @@ build_task:
> >         do
> >           $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1 ;
> >         done
> > +    - find . -not -path 'meson-logs/*' -delete
> 
> I'm not sure, but this might cause trouble if you run the Cirrus-CI job with
> terminal access for testing the binaries manually after the build succeeded?
> Maybe it would be better to just kill the .o files and leave the rest
> around?

Hmm, awkward, yes. I'll try and finese this a bit to just .o files and
see how it works out.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|