Re: [PATCH for-7.0] gitlab-ci: Add cirrus-ci based tests for NetBSD and OpenBSD

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Dec 09, 2021 at 11:31:24AM +0100, Thomas Huth wrote:
> Cirrus-CI provides KVM in their Linux containers, so we can also run
> our VM-based NetBSD and OpenBSD build jobs there.
> Since the VM installation might take a while, we only run the "help"
> target on the first invocation to avoid timeouts, and then only check
> the build during the next run, once the base image has been cached.
> For the the build tests, we also only use very a limited set of target
> CPUs since compiling in these VMs is not very fast (especially the
> build on OpenBSD seems to be incredibly slow).
> For the time being, the jobs are also marked as manually only, since
> this double-indirect setup (with the cirrus-run script and VMs in
> the Cirrus-CI containers) might fail more often than the other jobs.

I think they'll have to be manual forever basically unless
something changes in cirrus.

Historically we've had trouble with the cirrus jobs timing out.
This was ultimately a result of the fact that only 2 cirrus jobs
can run concurrently, and we had duplicate jobs being scheduled
on 'master' and 'staging'. This resulted in 4 jobs being queued
and most of the time, and because each job took > 30 minutes,
two of them would frequently hit the gitlab job 1 hour timeout.

Unless we can ensure that /all/ our cirrus jobs will reliably
completed in about 20 minutes in normal case (30 mins if
cirrus is being slow), then we can't have more than 2 cirrus
jobs as one or more will end up going over the 1 hour cutoff.

The idea of having NetBSD/OpenBSD jobs is good, but I think
it feels like a case where we're going to need to look at
using custom runners if we want them triggere on 'staging'.

Manual jobs could be ok for contributors forks at most.

> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  If you're interested, you can see some runs here:
>  NetBSD: https://gitlab.com/thuth/qemu/-/jobs/1865965079
>  OpenBSD: https://gitlab.com/thuth/qemu/-/jobs/1866116634
> 
>  .gitlab-ci.d/cirrus.yml           | 35 +++++++++++++++++++++++++++++++
>  .gitlab-ci.d/cirrus/kvm-build.yml | 31 +++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+)
>  create mode 100644 .gitlab-ci.d/cirrus/kvm-build.yml
> 
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> index 90bbe9f8ff..f2d4c0ca4f 100644
> --- a/.gitlab-ci.d/cirrus.yml
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -64,3 +64,38 @@ x64-macos-11-base-build:
>      PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
>      PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
>      TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> +
> +
> +# The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job
> +.cirrus_kvm_job:
> +  stage: build
> +  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> +  needs: []
> +  timeout: 80m
> +  allow_failure: true
> +  script:
> +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> +          -e "s|[@]NAME@|$NAME|g"
> +          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> +          -e "s|[@]TEST_TARGETS@|$TEST_TARGETS|g"
> +      <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> +    - cat .gitlab-ci.d/cirrus/$NAME.yml
> +    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> +  rules:
> +    - when: manual
> +
> +x86-netbsd:
> +  extends: .cirrus_kvm_job
> +  variables:
> +    NAME: netbsd
> +    CONFIGURE_ARGS: --target-list=x86_64-softmmu,ppc64-softmmu,aarch64-softmmu
> +    TEST_TARGETS: check
> +
> +x86-openbsd:
> +  extends: .cirrus_kvm_job
> +  variables:
> +    NAME: openbsd
> +    CONFIGURE_ARGS: --target-list=i386-softmmu,riscv64-softmmu,mips64-softmmu
> +    TEST_TARGETS: check
> diff --git a/.gitlab-ci.d/cirrus/kvm-build.yml b/.gitlab-ci.d/cirrus/kvm-build.yml
> new file mode 100644
> index 0000000000..4334fabf39
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/kvm-build.yml
> @@ -0,0 +1,31 @@
> +container:
> +  image: fedora:35
> +  cpu: 4
> +  memory: 8Gb
> +  kvm: true
> +
> +env:
> +  CIRRUS_CLONE_DEPTH: 1
> +  CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> +  CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> +  CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> +
> +@NAME@_task:
> +  @NAME@_vm_cache:
> +    folder: $HOME/.cache/qemu-vm
> +  install_script:
> +    - dnf update -y
> +    - dnf install -y git make openssh-clients qemu-img qemu-system-x86 wget
> +  clone_script:
> +    - git clone --depth 100 "$CI_REPOSITORY_URL" .
> +    - git fetch origin "$CI_COMMIT_REF_NAME"
> +    - git reset --hard "$CI_COMMIT_SHA"
> +  build_script:
> +    - if [ -f $HOME/.cache/qemu-vm/images/@NAME@.img ]; then
> +        make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN)
> +          EXTRA_CONFIGURE_OPTS="@CONFIGURE_ARGS@"
> +          BUILD_TARGET="@TEST_TARGETS@" ;
> +      else
> +        make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN) BUILD_TARGET=help
> +          EXTRA_CONFIGURE_OPTS="--disable-system --disable-user --disable-tools" ;
> +      fi
> -- 
> 2.27.0
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|