[PATCH 0/2] gitlab-ci: Make separate sections for build and test

[PATCH 0/2] gitlab-ci: Make separate sections for build and test

[Peter Maydell]

This patchset uses gitlab's support for putting special lines
in the log output that it will interpret as the beginning and
end markers for a collapsible section in the build log.

That's documented here, but it's just "write a line with
the right format and a timestamp and a magic escape sequence
in it":
https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections

Since gitlab puts little "time taken" bubbles for each
collapsible section, we can use this to easily find out how
much time is taken for build versus test.

Patch 1 splits out "make" and "make check" to separate make
invocations for the cross-build templates; we do this for
the native-build already.

Patch 2 adds the shell functions that emit start and end
section markers, and puts in section lines in the job
templates.

Since this is just for convenience in looking at CI job logs,
I have not put in a lot of effort into dividing things into
sections -- I've mostly just put configure/build/test into
the templates. We can always tweak the sections later if
we find out it would be useful.

If you want to see what the resulting logs look like you
can look at the jobs in this pipeline:
https://gitlab.com/pm215/qemu/-/pipelines/1458593312

thanks
-- PMM

Peter Maydell (2):
  .gitlab-ci.d: Split build and test in cross build job templates
  .gitlab-ci.d: Make separate collapsible log sections for build and
    test

 .gitlab-ci.d/buildtest-template.yml  | 14 ++++++++
 .gitlab-ci.d/buildtest.yml           |  1 +
 .gitlab-ci.d/crossbuild-template.yml | 48 ++++++++++++++++++++++++++--
 scripts/ci/gitlab-ci-section         | 29 +++++++++++++++++
 4 files changed, 89 insertions(+), 3 deletions(-)
 create mode 100644 scripts/ci/gitlab-ci-section

-- 
2.34.1

[PATCH 1/2] .gitlab-ci.d: Split build and test in cross build job templates

[Peter Maydell]

In the native_build_job_template we have separate steps in the script
for the build and the test steps.  This is helpful because then
gitlab will give separate timestamps in the log view for each, and
you can see how long it took to compile vs how long to test.  In the
templates in crossbuild-template.yml, however, we do both the build
and test in a single 'make' invocation, and so we don't get the
separate timing information.

Split the build and test, in the same way we do in the native build
template.

This will also give us a place to separate out how parallel we want
to do the build by default from how parallel we want to do the tests
by default, which might be helpful in future.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
Gitlab seems to be inconsistent about whether you get the
separate time-taken bubbles for each line of the script section
or not; but certainly if we don't split into separate 'make'
invocations we definitely can't get the timing info...
---
 .gitlab-ci.d/crossbuild-template.yml | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 2ce0432eb77..d1cb7a35dbf 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -9,6 +9,7 @@
     when: always
   timeout: 80m
   before_script:
+    - JOBS=$(expr $(nproc) + 1)
     - cat /packages.txt
   script:
     - export CCACHE_BASEDIR="$(pwd)"
@@ -24,7 +25,11 @@
           i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
           mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
           sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS"
-    - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+    - make -j"$JOBS" all check-build
+    - if test -n "$MAKE_CHECK_ARGS";
+      then
+        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
+      fi
     - if grep -q "EXESUF=.exe" config-host.mak;
       then make installer;
       version="$(git describe --match v[0-9]* 2>/dev/null || git rev-parse --short HEAD)";
@@ -46,6 +51,8 @@
     paths:
       - ccache/
     key: "$CI_JOB_NAME"
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
@@ -55,7 +62,11 @@
     - cd build
     - ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
         --disable-tools --enable-${ACCEL:-kvm} $EXTRA_CONFIGURE_OPTS
-    - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+    - make -j"$JOBS" all check-build
+    - if test -n "$MAKE_CHECK_ARGS";
+      then
+        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
+      fi
 
 .cross_user_build_job:
   extends: .base_job_template
@@ -65,6 +76,8 @@
     paths:
       - ccache/
     key: "$CI_JOB_NAME"
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
@@ -76,7 +89,11 @@
           alpha-linux-user m68k-linux-user microblazeel-linux-user
           or1k-linux-user ppc-linux-user sparc-linux-user
           xtensa-linux-user $CROSS_SKIP_TARGETS"
-    - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+    - make -j"$JOBS" all check-build
+    - if test -n "$MAKE_CHECK_ARGS";
+      then
+        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
+      fi
 
 # We can still run some tests on some of our cross build jobs. They can add this
 # template to their extends to save the build logs and test results
-- 
2.34.1

[PATCH 2/2] .gitlab-ci.d: Make separate collapsible log sections for build and test

[Peter Maydell]

GitLab lets a CI job create its own collapsible log sections by
emitting special escape codes, as documented here:

https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections

Use these to make "configure", "build" and "test" separate
collapsible stages.

As recommended by the GitLab docs, we use some shell which is
sourced in the CI job to define functions to emit the magic
lines that start and end sections, to hide the ugliness of
the printf lines from the log.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 .gitlab-ci.d/buildtest-template.yml  | 14 ++++++++++++++
 .gitlab-ci.d/buildtest.yml           |  1 +
 .gitlab-ci.d/crossbuild-template.yml | 25 ++++++++++++++++++++++++
 scripts/ci/gitlab-ci-section         | 29 ++++++++++++++++++++++++++++
 4 files changed, 69 insertions(+)
 create mode 100644 scripts/ci/gitlab-ci-section

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 5f2fc7e6f49..8c69c60d215 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -8,8 +8,11 @@
     key: "$CI_JOB_NAME"
     when: always
   before_script:
+    - source scripts/ci/gitlab-ci-section
+    - section_start setup "Pre-script setup"
     - JOBS=$(expr $(nproc) + 1)
     - cat /packages.txt
+    - section_end setup
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
@@ -19,6 +22,7 @@
     - mkdir build
     - cd build
     - ccache --zero-stats
+    - section_start configure "Running configure"
     - ../configure --enable-werror --disable-docs --enable-fdt=system
           ${TARGETS:+--target-list="$TARGETS"}
           $CONFIGURE_ARGS ||
@@ -27,11 +31,16 @@
       then
         pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
+    - section_end configure
+    - section_start build "Building QEMU"
     - $MAKE -j"$JOBS"
+    - section_end build
+    - section_start test "Running tests"
     - if test -n "$MAKE_CHECK_ARGS";
       then
         $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+    - section_end test
     - ccache --show-stats
 
 # We jump some hoops in common_test_job_template to avoid
@@ -54,6 +63,8 @@
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   script:
+    - source scripts/ci/gitlab-ci-section
+    - section_start buildenv "Setting up to run tests"
     - scripts/git-submodule.sh update roms/SLOF
     - meson subprojects download $(cd build/subprojects && echo *)
     - cd build
@@ -63,7 +74,10 @@
     - if [ "x${QEMU_TEST_CACHE_DIR}" != "x" ]; then
         $MAKE precache-functional ;
       fi
+    - section_end buildenv
+    - section_start test "Running tests"
     - $MAKE NINJA=":" $MAKE_CHECK_ARGS
+    - section_end test
 
 .native_test_job_template:
   extends: .common_test_job_template
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 2ab8c4806e4..87848c2ffe8 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -188,6 +188,7 @@ build-previous-qemu:
     # Override the default flags as we need more to grab the old version
     GIT_FETCH_EXTRA_FLAGS: --prune --quiet
   before_script:
+    - source scripts/ci/gitlab-ci-section
     - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)"
     - git remote add upstream https://gitlab.com/qemu-project/qemu
     - git fetch upstream refs/tags/$QEMU_PREV_VERSION:refs/tags/$QEMU_PREV_VERSION
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index d1cb7a35dbf..45a98103554 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -9,8 +9,11 @@
     when: always
   timeout: 80m
   before_script:
+    - source scripts/ci/gitlab-ci-section
+    - section_start setup "Pre-script setup"
     - JOBS=$(expr $(nproc) + 1)
     - cat /packages.txt
+    - section_end setup
   script:
     - export CCACHE_BASEDIR="$(pwd)"
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
@@ -19,22 +22,30 @@
     - mkdir build
     - cd build
     - ccache --zero-stats
+    - section_start configure "Running configure"
     - ../configure --enable-werror --disable-docs --enable-fdt=system
         --disable-user $QEMU_CONFIGURE_OPTS $EXTRA_CONFIGURE_OPTS
         --target-list-exclude="arm-softmmu
           i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
           mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
           sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS"
+    - section_end configure
+    - section_start build "Building QEMU"
     - make -j"$JOBS" all check-build
+    - section_end build
+    - section_start test "Running tests"
     - if test -n "$MAKE_CHECK_ARGS";
       then
         $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+    - section_end test
+    - section_start installer "Building the installer"
     - if grep -q "EXESUF=.exe" config-host.mak;
       then make installer;
       version="$(git describe --match v[0-9]* 2>/dev/null || git rev-parse --short HEAD)";
       mv -v qemu-setup*.exe qemu-setup-${version}.exe;
       fi
+    - section_end installer
     - ccache --show-stats
 
 # Job to cross-build specific accelerators.
@@ -52,6 +63,7 @@
       - ccache/
     key: "$CI_JOB_NAME"
   before_script:
+    - source scripts/ci/gitlab-ci-section
     - JOBS=$(expr $(nproc) + 1)
   script:
     - export CCACHE_BASEDIR="$(pwd)"
@@ -60,13 +72,19 @@
     - export PATH="$CCACHE_WRAPPERSDIR:$PATH"
     - mkdir build
     - cd build
+    - section_start configure "Running configure"
     - ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
         --disable-tools --enable-${ACCEL:-kvm} $EXTRA_CONFIGURE_OPTS
+    - section_end configure
+    - section_start build "Building QEMU"
     - make -j"$JOBS" all check-build
+    - section_end build
+    - section_start test "Running tests"
     - if test -n "$MAKE_CHECK_ARGS";
       then
         $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+    - section_end test
 
 .cross_user_build_job:
   extends: .base_job_template
@@ -77,6 +95,7 @@
       - ccache/
     key: "$CI_JOB_NAME"
   before_script:
+    - source scripts/ci/gitlab-ci-section
     - JOBS=$(expr $(nproc) + 1)
   script:
     - export CCACHE_BASEDIR="$(pwd)"
@@ -84,16 +103,22 @@
     - export CCACHE_MAXSIZE="500M"
     - mkdir build
     - cd build
+    - section_start configure "Running configure"
     - ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
         --disable-system --target-list-exclude="aarch64_be-linux-user
           alpha-linux-user m68k-linux-user microblazeel-linux-user
           or1k-linux-user ppc-linux-user sparc-linux-user
           xtensa-linux-user $CROSS_SKIP_TARGETS"
+    - section_end configure
+    - section_start build "Building QEMU"
     - make -j"$JOBS" all check-build
+    - section_end build
+    - section_start test "Running tests"
     - if test -n "$MAKE_CHECK_ARGS";
       then
         $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+    - section_end test
 
 # We can still run some tests on some of our cross build jobs. They can add this
 # template to their extends to save the build logs and test results
diff --git a/scripts/ci/gitlab-ci-section b/scripts/ci/gitlab-ci-section
new file mode 100644
index 00000000000..9bbe80420d6
--- /dev/null
+++ b/scripts/ci/gitlab-ci-section
@@ -0,0 +1,29 @@
+# Copyright (c) 2024 Linaro Ltd
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+# gitlab-ci-section: This is a shell script fragment which defines
+# functions section_start and section_end which will emit marker lines
+# that GitLab will interpret as the beginning or end of a "collapsible
+# section" in a CI job log. See
+# https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections
+#
+# This is intended to be sourced in the before_script section of
+# a CI config; the section_start and section_end functions will
+# then be available for use in the before_script and script sections.
+
+# Section names are [-_.A-Za-z0-9] and the section_start pairs with
+# a section_end with the same section name.
+# The description can be any printable text without newlines; this is
+# what will appear in the log.
+
+# Usage:
+# section_start section_name "Description of the section"
+section_start () {
+    printf "section_start:%s:%s\r\e[0K%s\n" "$(date +%s)" "$1" "$2"
+}
+
+# Usage:
+# section_end section_name
+section_end () {
+    printf "section_end:%s:%s\r\e[0K\n" "$(date +%s)" "$1"
+}
-- 
2.34.1

Re: [PATCH 0/2] gitlab-ci: Make separate sections for build and test

[Richard Henderson]

On 9/18/24 14:54, Peter Maydell wrote:
> Peter Maydell (2):
>    .gitlab-ci.d: Split build and test in cross build job templates
>    .gitlab-ci.d: Make separate collapsible log sections for build and
>      test

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [PATCH 1/2] .gitlab-ci.d: Split build and test in cross build job templates

[Thomas Huth]

On 18/09/2024 14.54, Peter Maydell wrote:
> In the native_build_job_template we have separate steps in the script
> for the build and the test steps.  This is helpful because then
> gitlab will give separate timestamps in the log view for each, and
> you can see how long it took to compile vs how long to test.  In the
> templates in crossbuild-template.yml, however, we do both the build
> and test in a single 'make' invocation, and so we don't get the
> separate timing information.
> 
> Split the build and test, in the same way we do in the native build
> template.
> 
> This will also give us a place to separate out how parallel we want
> to do the build by default from how parallel we want to do the tests
> by default, which might be helpful in future.
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/2] .gitlab-ci.d: Make separate collapsible log sections for build and test

[Thomas Huth]

On 18/09/2024 14.54, Peter Maydell wrote:
> GitLab lets a CI job create its own collapsible log sections by
> emitting special escape codes, as documented here:
> 
> https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections
> 
> Use these to make "configure", "build" and "test" separate
> collapsible stages.
> 
> As recommended by the GitLab docs, we use some shell which is
> sourced in the CI job to define functions to emit the magic
> lines that start and end sections, to hide the ugliness of
> the printf lines from the log.
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---

Reviewed-by: Thomas Huth <thuth@redhat.com>