Re: [RFC PATCH-for-8.2] .gitlab-ci.d/cirrus.yml: Promote NetBSD job as gating

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Nov 09, 2023 at 04:35:56PM +0100, Philippe Mathieu-Daudé wrote:
> On 9/11/23 16:35, Philippe Mathieu-Daudé wrote:
> > This Cirrus-CI based job takes ~12min, similarly to macOS job.
> > 
> > Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> > ---
> > Based-on: <20231109150900.91186-1-philmd@linaro.org>
> >            "tests/vm/netbsd: Use Python v3.11"
> > ---
> >   .gitlab-ci.d/cirrus.yml | 3 +--
> >   1 file changed, 1 insertion(+), 2 deletions(-)
> > 
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > index e7f1f83c2c..7b01acb104 100644
> > --- a/.gitlab-ci.d/cirrus.yml
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -94,8 +94,6 @@ aarch64-macos-12-base-build:
> >       - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> >     variables:
> >       QEMU_JOB_CIRRUS: 1
> > -    QEMU_JOB_OPTIONAL: 1
> > -
> >   x86-netbsd:
> >     extends: .cirrus_kvm_job
> > @@ -110,3 +108,4 @@ x86-openbsd:
> >       NAME: openbsd
> >       CONFIGURE_ARGS: --target-list=i386-softmmu,riscv64-softmmu,mips64-softmmu
> >       TEST_TARGETS: check
> > +    QEMU_JOB_OPTIONAL: 1
> 
> BTW OpenBSD works for me, but takes ~20min (similar to the FreeBSD job).

In the most recent pipeline FreeBSD too 22 mins, macOS 14 mins.

The key factor is that the Cirrus job needs to complete before the
GitLab job times out. We have a 1 hr 20 limit on GitLab jobs, and
Cirrus CI allows 2 jobs in parallel.

So in the worst case where the OpenBSD job was blocked until FreeBSD
job finished, we would be waiting about 45 mins for completion.

In the best case we would be waiting 36 mins.

Well technically the worst case would be no parallelization at all,
which is 70 mins serialized execution time, which is kinda close to
the 1hr20 limit. This does sometimes happen, but I don't know how
often.

Sometimes Cirrus CI can also delay the jobs starting for a while
due to lack of runners.

IOW, in normal times we could afford to enable all these jobs, but
if Cirrus CI is under heavy load we increase chance of timeouts.

I could have sworn our cirrus jobs were much slower in the past (around
the 40 min mark), as we had to cut down what we were running to avoid
frequent timeouts.

I'd say lets wait until this release is done though. Enable OpenBSD/NetBSD
when the new dev cycle opens up, so we can watch their stablility for a
bit, and not impact stability of the stable branch for this forthcoming
release.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|