* [PATCH 0/1] gitlab: enable ccache for many build jobs
@ 2023-08-04 11:10 Daniel P. Berrangé
2023-08-04 11:10 ` [PATCH 1/1] " Daniel P. Berrangé
2023-08-10 15:29 ` [PATCH 0/1] " Alex Bennée
0 siblings, 2 replies; 3+ messages in thread
From: Daniel P. Berrangé @ 2023-08-04 11:10 UTC (permalink / raw)
To: qemu-devel
Cc: Wainer dos Santos Moschetta, Beraldo Leal, Thomas Huth,
Yonggang Luo, Alex Bennée, Philippe Mathieu-Daudé,
Daniel P. Berrangé
Thomas has previously proposed adding ccache:
https://lists.nongnu.org/archive/html/qemu-devel/2021-04/msg02189.html
This is an pretty much suggesting the same.
In jobs which are compile heavy the results can be pretty
spectacular. Many jobs get their execution time cut by as
much as 70%, a few even get over 80% saving.
The 'build-XXXX' and 'cross-XXXX' jobs are what benefit.
The 'check-XXXX' jobs won't benefit much as they're not compiling
stuff and likewise the 'container-xxxx' jobs.
Overall we get a 35% reduction in CI minutes consumed.
NB this does not directly correspond to the same wallclock time
reduction, since we're only improving the 'build' stage performance.
Given CI quota controls reducing CI minutes consumed is beneficial
even if wallclock time is unchanged.
When the cache is cold, however, there is some degradation of
performance. The biggest issue was/is the msys jobs, which
had to be mitigated through ccache tuning.
The breakdown in my test pipelines between totally empty
cache and (what ought to be) a 100% hit rate for cache:
msys2-64bit: 61.51% saving (4159 secs -> 1601 secs)
check-python-minreqs: ( 109 secs -> 104 secs)
pages: ( 437 secs -> 481 secs)
check-cfi-x86_64: ( 837 secs -> 880 secs)
check-system-opensuse: (1351 secs -> 1342 secs)
check-system-centos: ( 922 secs -> 945 secs)
crash-test-fedora: ( 599 secs -> 577 secs)
check-system-fedora: ( 770 secs -> 799 secs)
crash-test-debian: (1489 secs -> 1499 secs)
check-system-debian: (1342 secs -> 1314 secs)
check-system-ubuntu: ( 608 secs -> 604 secs)
check-system-alpine: ( 298 secs -> 297 secs)
msys2-32bit: 51.55% saving (3486 secs -> 1689 secs)
aarch64-macos-12-base-build: (1050 secs -> 973 secs)
x64-freebsd-13-build: (1417 secs -> 1452 secs)
check-dco: ( 35 secs -> 33 secs)
check-patch: ( 34 secs -> 34 secs)
build-tools-and-docs-debian: 30.13% saving ( 395 secs -> 276 secs)
build-libvhost-user: ( 122 secs -> 123 secs)
build-without-defaults: 34.01% saving ( 691 secs -> 456 secs)
build-tci: (1199 secs -> 1187 secs)
build-oss-fuzz: ( 880 secs -> 909 secs)
gcov: 68.24% saving (1206 secs -> 383 secs)
tsan-build: 77.12% saving (1333 secs -> 305 secs)
build-cfi-x86_64: 10.10% saving (2625 secs -> 2360 secs)
clang-user: (1850 secs -> 1797 secs)
clang-system: 66.36% saving (1605 secs -> 540 secs)
build-tricore-softmmu: ( 228 secs -> 231 secs)
build-some-softmmu: ( 519 secs -> 526 secs)
build-user-hexagon: 44.66% saving ( 309 secs -> 171 secs)
build-user-static: (1338 secs -> 1298 secs)
build-user: (1552 secs -> 1600 secs)
build-tcg-disabled: ( 879 secs -> 819 secs)
build-system-opensuse: 55.97% saving (1088 secs -> 479 secs)
build-system-centos: (1053 secs -> 1056 secs)
build-system-fedora: 69.61% saving (1244 secs -> 378 secs)
build-system-debian: 72.69% saving (1095 secs -> 299 secs)
build-system-ubuntu: 63.69% saving ( 851 secs -> 309 secs)
build-system-alpine: 71.24% saving (1109 secs -> 319 secs)
cross-arm64-xen-only: 76.59% saving ( 786 secs -> 184 secs)
cross-amd64-xen-only: 73.49% saving ( 679 secs -> 180 secs)
cross-win64-system: 71.48% saving (2952 secs -> 842 secs)
cross-win32-system: 56.86% saving (2406 secs -> 1038 secs)
cross-mips64el-kvm-only: 72.67% saving ( 611 secs -> 167 secs)
cross-s390x-kvm-only: 67.55% saving ( 527 secs -> 171 secs)
cross-s390x-user: 14.92% saving (1146 secs -> 975 secs)
cross-s390x-system: 83.31% saving (1989 secs -> 332 secs)
cross-riscv64-user: ( 165 secs -> 170 secs)
cross-riscv64-system: 82.25% saving (2118 secs -> 376 secs)
cross-ppc64el-kvm-only: 68.51% saving ( 578 secs -> 182 secs)
cross-ppc64el-user: (1113 secs -> 1063 secs)
cross-ppc64el-system: 82.69% saving (2062 secs -> 357 secs)
cross-mips64el-user: (1201 secs -> 1258 secs)
cross-mips64el-system: 81.12% saving (2023 secs -> 382 secs)
cross-mipsel-user: (1232 secs -> 1223 secs)
cross-mipsel-system: 83.04% saving (2093 secs -> 355 secs)
cross-i386-tci: (1358 secs -> 1333 secs)
cross-i386-user: (1069 secs -> 1182 secs)
cross-arm64-kvm-only: 66.25% saving ( 480 secs -> 162 secs)
cross-arm64-user: (1126 secs -> 1097 secs)
cross-arm64-system: 84.11% saving (2202 secs -> 350 secs)
cross-armhf-user: (1101 secs -> 1162 secs)
cross-armel-user: (1096 secs -> 1173 secs)
python-container: ( 136 secs -> 137 secs)
amd64-opensuse-leap-container: ( 146 secs -> 146 secs)
amd64-ubuntu2204-container: ( 142 secs -> 138 secs)
amd64-debian-container: ( 144 secs -> 146 secs)
amd64-alpine-container: ( 134 secs -> 131 secs)
win64-fedora-cross-container: ( 166 secs -> 160 secs)
win32-fedora-cross-container: ( 160 secs -> 163 secs)
i386-fedora-cross-container: ( 133 secs -> 121 secs)
cris-fedora-cross-container: ( 113 secs -> 114 secs)
xtensa-debian-cross-container: ( 116 secs -> 117 secs)
tricore-debian-cross-container: ( 120 secs -> 124 secs)
sparc64-debian-cross-container: ( 107 secs -> 108 secs)
sh4-debian-cross-container: ( 103 secs -> 104 secs)
s390x-debian-cross-container: ( 145 secs -> 139 secs)
riscv64-debian-test-cross-container: ( 106 secs -> 108 secs)
riscv64-debian-cross-container: ( 128 secs -> 128 secs)
ppc64el-debian-cross-container: ( 141 secs -> 148 secs)
powerpc-test-cross-container: ( 120 secs -> 119 secs)
mipsel-debian-cross-container: ( 137 secs -> 136 secs)
mips-debian-cross-container: ( 109 secs -> 106 secs)
mips64el-debian-cross-container: ( 145 secs -> 139 secs)
mips64-debian-cross-container: ( 105 secs -> 105 secs)
m68k-debian-cross-container: ( 106 secs -> 105 secs)
hppa-debian-cross-container: ( 106 secs -> 104 secs)
hexagon-cross-container: ( 153 secs -> 151 secs)
armhf-debian-cross-container: ( 146 secs -> 143 secs)
armel-debian-cross-container: ( 140 secs -> 147 secs)
arm64-debian-cross-container: ( 143 secs -> 146 secs)
amd64-debian-user-cross-container: ( 169 secs -> 165 secs)
amd64-debian-cross-container: ( 135 secs -> 133 secs)
alpha-debian-cross-container: ( 105 secs -> 106 secs)
amd64-fedora-container: ( 159 secs -> 156 secs)
amd64-centos8-container: ( 146 secs -> 148 secs)
Total 35.74% saving (78591 secs -> 50500 secs)
Daniel P. Berrangé (1):
gitlab: enable ccache for many build jobs
.gitlab-ci.d/buildtest-template.yml | 11 ++++++++
.gitlab-ci.d/crossbuild-template.yml | 26 +++++++++++++++++++
.gitlab-ci.d/windows.yml | 13 ++++++++--
docs/devel/ci-jobs.rst.inc | 7 +++++
.../dockerfiles/debian-hexagon-cross.docker | 9 ++++++-
5 files changed, 63 insertions(+), 3 deletions(-)
--
2.41.0
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH 1/1] gitlab: enable ccache for many build jobs
2023-08-04 11:10 [PATCH 0/1] gitlab: enable ccache for many build jobs Daniel P. Berrangé
@ 2023-08-04 11:10 ` Daniel P. Berrangé
2023-08-10 15:29 ` [PATCH 0/1] " Alex Bennée
1 sibling, 0 replies; 3+ messages in thread
From: Daniel P. Berrangé @ 2023-08-04 11:10 UTC (permalink / raw)
To: qemu-devel
Cc: Wainer dos Santos Moschetta, Beraldo Leal, Thomas Huth,
Yonggang Luo, Alex Bennée, Philippe Mathieu-Daudé,
Daniel P. Berrangé
The `ccache` tool can be very effective at reducing compilation times
when re-running pipelines with only minor changes each time. For example
a fresh 'build-system-fedora' job will typically take 20 minutes on the
gitlab.com shared runners. With ccache this is reduced to as little as
6 minutes.
Normally meson would auto-detect existance of ccache in $PATH and use
it automatically, but the way we wrap meson from configure breaks this,
as we're passing in an config file with explicitly set compiler paths.
Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For
unknown reasons if doing this in msys though, gcc becomes unable to
invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc'
before invoking 'configure' instead.
A second problem with msys is that cache misses are incredibly
expensive, so enabling ccache massively slows down the build when
the cache isn't well populated. This is suspected to be a result of
the cost of spawning processes under the msys architecture. To deal
with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only'
strategy. This avoids extra spawning of the pre-processor during
cache misses, with the downside that is it less likely ccache will
find a cache hit after semantically benign compiler flag changes.
This is the lesser of two evils, as otherwise we can't use ccache
at all under msys and remain inside the job time limit.
If people are finding ccache to hurt their pipelines, it can be
disabled by setting the 'CCACHE_DISABLE=1' env variable against
their gitlab fork CI settings.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/buildtest-template.yml | 11 ++++++++
.gitlab-ci.d/crossbuild-template.yml | 26 +++++++++++++++++++
.gitlab-ci.d/windows.yml | 13 ++++++++--
docs/devel/ci-jobs.rst.inc | 7 +++++
.../dockerfiles/debian-hexagon-cross.docker | 9 ++++++-
5 files changed, 63 insertions(+), 3 deletions(-)
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index f3e39b7eb1..4fbfeb6667 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -2,11 +2,21 @@
extends: .base_job_template
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
+ cache:
+ paths:
+ - ccache
+ key: "$CI_JOB_NAME"
+ when: always
before_script:
- JOBS=$(expr $(nproc) + 1)
script:
+ - export CCACHE_BASEDIR="$(pwd)"
+ - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
+ - export CCACHE_MAXSIZE="500M"
+ - export PATH="$CCACHE_WRAPPERSDIR:$PATH"
- mkdir build
- cd build
+ - ccache --zero-stats
- ../configure --enable-werror --disable-docs --enable-fdt=system
${TARGETS:+--target-list="$TARGETS"}
$CONFIGURE_ARGS ||
@@ -20,6 +30,7 @@
then
make -j"$JOBS" $MAKE_CHECK_ARGS ;
fi
+ - ccache --show-stats
# We jump some hoops in common_test_job_template to avoid
# rebuilding all the object files we skip in the artifacts
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index d97611053b..3e5f4d9cd8 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -2,10 +2,20 @@
extends: .base_job_template
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
+ cache:
+ paths:
+ - ccache
+ key: "$CI_JOB_NAME"
+ when: always
timeout: 80m
script:
+ - export CCACHE_BASEDIR="$(pwd)"
+ - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
+ - export CCACHE_MAXSIZE="500M"
+ - export PATH="$CCACHE_WRAPPERSDIR:$PATH"
- mkdir build
- cd build
+ - ccache --zero-stats
- ../configure --enable-werror --disable-docs --enable-fdt=system
--disable-user $QEMU_CONFIGURE_OPTS $EXTRA_CONFIGURE_OPTS
--target-list-exclude="arm-softmmu cris-softmmu
@@ -18,6 +28,7 @@
version="$(git describe --match v[0-9]* 2>/dev/null || git rev-parse --short HEAD)";
mv -v qemu-setup*.exe qemu-setup-${version}.exe;
fi
+ - ccache --show-stats
# Job to cross-build specific accelerators.
#
@@ -29,7 +40,15 @@
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
timeout: 30m
+ cache:
+ paths:
+ - ccache/
+ key: "$CI_JOB_NAME"
script:
+ - export CCACHE_BASEDIR="$(pwd)"
+ - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
+ - export CCACHE_MAXSIZE="500M"
+ - export PATH="$CCACHE_WRAPPERSDIR:$PATH"
- mkdir build
- cd build
- ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
@@ -40,7 +59,14 @@
extends: .base_job_template
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
+ cache:
+ paths:
+ - ccache/
+ key: "$CI_JOB_NAME"
script:
+ - export CCACHE_BASEDIR="$(pwd)"
+ - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
+ - export CCACHE_MAXSIZE="500M"
- mkdir build
- cd build
- ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index cd7622a761..12a987cd71 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -5,13 +5,14 @@
- windows
- windows-1809
cache:
- key: "${CI_JOB_NAME}-cache"
+ key: "$CI_JOB_NAME"
paths:
- msys64/var/cache
+ - ccache
when: always
needs: []
stage: build
- timeout: 80m
+ timeout: 100m
variables:
# This feature doesn't (currently) work with PowerShell, it stops
# the echo'ing of commands being run and doesn't show any timing
@@ -72,6 +73,7 @@
bison diffutils flex
git grep make sed
$MINGW_TARGET-capstone
+ $MINGW_TARGET-ccache
$MINGW_TARGET-curl
$MINGW_TARGET-cyrus-sasl
$MINGW_TARGET-dtc
@@ -101,11 +103,18 @@
- Write-Output "Running build at $(Get-Date -Format u)"
- $env:CHERE_INVOKING = 'yes' # Preserve the current working directory
- $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
+ - $env:CCACHE_BASEDIR = "$env:CI_PROJECT_DIR"
+ - $env:CCACHE_DIR = "$env:CCACHE_BASEDIR/ccache"
+ - $env:CCACHE_MAXSIZE = "500M"
+ - $env:CCACHE_DEPEND = 1 # cache misses are too expensive with preprocessor mode
+ - $env:CC = "ccache gcc"
- mkdir build
- cd build
+ - ..\msys64\usr\bin\bash -lc "ccache --zero-stats"
- ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
- ..\msys64\usr\bin\bash -lc "make"
- ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
+ - ..\msys64\usr\bin\bash -lc "ccache --show-stats"
- Write-Output "Finished build at $(Get-Date -Format u)"
msys2-64bit:
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 3f6802d51e..4c39cdb2d9 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -188,3 +188,10 @@ If you've got access to a CentOS Stream 8 x86_64 host that can be
used as a gitlab-CI runner, you can set this variable to enable the
tests that require this kind of host. The runner should be tagged with
both "centos_stream_8" and "x86_64".
+
+CCACHE_DISABLE
+~~~~~~~~~~~~~~
+The jobs are configured to use "ccache" by default since this typically
+reduces compilation time, at the cost of increased storage. If the
+use of "ccache" is suspected to be hurting the overall job execution
+time, setting the "CCACHE_DISABLE=1" env variable to disable it.
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
index c2cfb6a5d0..578269766c 100644
--- a/tests/docker/dockerfiles/debian-hexagon-cross.docker
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -15,6 +15,7 @@ RUN apt-get update && \
# Install common build utilities
apt-get install -y --no-install-recommends \
curl \
+ ccache \
xz-utils \
ca-certificates \
bison \
@@ -24,13 +25,19 @@ RUN apt-get update && \
python3-venv && \
# Install QEMU build deps for use in CI
DEBIAN_FRONTEND=noninteractive eatmydata \
- apt build-dep -yy --arch-only qemu
+ apt build-dep -yy --arch-only qemu && \
+ mkdir -p /usr/libexec/ccache-wrappers && \
+ ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/c++ && \
+ ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \
+ ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
+ ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
ENV TOOLCHAIN_INSTALL /opt
ENV TOOLCHAIN_RELEASE 16.0.0
ENV TOOLCHAIN_BASENAME "clang+llvm-${TOOLCHAIN_RELEASE}-cross-hexagon-unknown-linux-musl"
ENV TOOLCHAIN_URL https://codelinaro.jfrog.io/artifactory/codelinaro-toolchain-for-hexagon/v${TOOLCHAIN_RELEASE}/${TOOLCHAIN_BASENAME}.tar.xz
+ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
RUN curl -#SL "$TOOLCHAIN_URL" | tar -xJC "$TOOLCHAIN_INSTALL"
ENV PATH $PATH:${TOOLCHAIN_INSTALL}/${TOOLCHAIN_BASENAME}/x86_64-linux-gnu/bin
--
2.41.0
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH 0/1] gitlab: enable ccache for many build jobs
2023-08-04 11:10 [PATCH 0/1] gitlab: enable ccache for many build jobs Daniel P. Berrangé
2023-08-04 11:10 ` [PATCH 1/1] " Daniel P. Berrangé
@ 2023-08-10 15:29 ` Alex Bennée
1 sibling, 0 replies; 3+ messages in thread
From: Alex Bennée @ 2023-08-10 15:29 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: qemu-devel, Wainer dos Santos Moschetta, Beraldo Leal,
Thomas Huth, Yonggang Luo, Philippe Mathieu-Daudé
Daniel P. Berrangé <berrange@redhat.com> writes:
> Thomas has previously proposed adding ccache:
>
> https://lists.nongnu.org/archive/html/qemu-devel/2021-04/msg02189.html
>
> This is an pretty much suggesting the same.
Queued to for-8.1/misc-fixes, thanks.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-08-10 15:29 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-08-04 11:10 [PATCH 0/1] gitlab: enable ccache for many build jobs Daniel P. Berrangé
2023-08-04 11:10 ` [PATCH 1/1] " Daniel P. Berrangé
2023-08-10 15:29 ` [PATCH 0/1] " Alex Bennée
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).