From: "Alex Bennée" <alex.bennee@linaro.org>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: "Brian Cain" <brian.cain@oss.qualcomm.com>,
"Pierrick Bouvier" <pierrick.bouvier@oss.qualcomm.com>,
qemu-devel@nongnu.org, "Stefan Hajnoczi" <stefanha@redhat.com>,
"Daniel P . Berrangé" <berrange@redhat.com>,
"Richard Henderson" <richard.henderson@linaro.org>,
"Philippe Mathieu-Daudé" <philmd@oss.qualcomm.com>,
"Paolo Bonzini" <pbonzini@redhat.com>
Subject: Re: [PATCH] ci: add automatic retry
Date: Wed, 21 Oct 2026 11:56:44 +0100 [thread overview]
Message-ID: <87fqxz1fv7.fsf@draig.linaro.org> (raw)
In-Reply-To: <CAFEAcA8-evgTa4GgvHV7HjUAeqnee1CN3onAK-EBH42saRqNag@mail.gmail.com> (Peter Maydell's message of "Wed, 21 Oct 2026 10:20:57 +0100")
Peter Maydell <peter.maydell@linaro.org> writes:
> On Wed, 21 Oct 2026 at 03:41, Brian Cain <brian.cain@oss.qualcomm.com> wrote:
>>
>>
>> On 10/20/2026 6:16 PM, Pierrick Bouvier wrote:
>> > When merging PR on the project, it's possible to observe failures on CI
>> > that can be related to sporadic test failures or infrastructure issues
>> > (pod eviction with Kubernetes). As a result, it adds latency on the
>> > process and workload for our top maintainers.
>> >
>> > This patch sets an automatic retry (supported by gitlab out of the box),
>> > so failing jobs are automatically retriggered.
>> >
>> > It's important to note that in case there is a real test or build
>> > failure, concerned jobs will run a second time, and fail again. That
>> > said, it happens seldomly since sub maintainers are supposed to test for
>> > this before sending it. Also, the retry is limited to the job that
>> > failed, so it should be cheap in terms of CI minutes.
>>
>> Will this effectively mask those flaky failures so that no one hears
>> about them? Is there a way to find out about them with something like a
>> weekly/monthly digest of failures?
>
> Broadly speaking, it's already the case that flaky failures are masked.
> Random infra-type failures / random timeouts /etc are common enough
> that when I'm doing a merge CI run I will just hit the retry button
> without much deep investigation.
>
> In an ideal world we'd pull down the gitlab job history and look
> through it for cases of "job X failed running against commit hash Y,
> but we also saw job X succeed on commit hash Y" (i.e. the retry
> worked second time) and then check the failure logs for what
> kinds of things are intermittent like that. But that requires
> somebody to put in the work to do that analysis. Gitlab's own
> CI job history web UI is very bad and doesn't let you do even
> basic things like "show me just the history for job X".
The gitlab-failure-analysis script was an attempt to start doing this,
so the last 10 failing staging pipelines where:
$ ./scripts/ci/gitlab-failure-analysis --count 10
failed pipeline 2949515618, total jobs 120, skipped 5, failed 8, 45059 tests, 1112 skipped tests, 8 failed tests
Failed job 17194356901, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17194356901
Failed job 17192659533, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17192659533
Failed job 17192659440, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17192659440
Failed job 17192659398, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17192659398
Failed job 17192659309, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17192659309
Failed job 17192659287, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17192659287
Failed job 17192659245, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17192659245
Failed job 17192659237, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17192659237
Failed test qemu.block-nbd-optional+block-thorough - qemu:io-nbd-nbd-stale-connection, block: [nbd], 1 raw nbd-stale-connection
Failed test qemu.block-parallels-optional+block-thorough - qemu:io-parallels-nbd-stale-connection, block: [parallels], 1 parallels nbd-stale-connection
Failed test qemu.block-qcow2-optional+block-slow - qemu:io-qcow2-nbd-stale-connection, block: [qcow2], 1 qcow2 nbd-stale-connection
Failed test qemu.block-qed-optional+block-thorough - qemu:io-qed-nbd-stale-connection, block: [qed], 1 qed nbd-stale-connection
Failed test qemu.block-vdi-optional+block-thorough - qemu:io-vdi-nbd-stale-connection, block: [vdi], 1 vdi nbd-stale-connection
Failed test qemu.block-vhdx-optional+block-thorough - qemu:io-vhdx-nbd-stale-connection, block: [vhdx], 1 vhdx nbd-stale-connection
Failed test qemu.block-vmdk-optional+block-thorough - qemu:io-vmdk-nbd-stale-connection, block: [vmdk], 1 vmdk nbd-stale-connection
Failed test qemu.block-luks-optional+block-thorough - qemu:io-luks-nbd-stale-connection, block: [luks], 1 luks nbd-stale-connection
failed pipeline 2945631627, total jobs 120, skipped 5, failed 13, 0 tests, 0 skipped tests, 0 failed tests
Failed job 17162982778, check-python-minreqs, https://gitlab.com/qemu-project/qemu/-/jobs/17162982778
Failed job 17162982652, block: [vpc], https://gitlab.com/qemu-project/qemu/-/jobs/17162982652
Failed job 17162982622, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17162982622
Failed job 17162982605, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17162982605
Failed job 17162982484, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17162982484
Failed job 17162982417, block: [raw], https://gitlab.com/qemu-project/qemu/-/jobs/17162982417
Failed job 17162982413, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17162982413
Failed job 17162982411, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17162982411
Failed job 17162982380, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17162982380
Failed job 17162982361, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17162982361
Failed job 17162982339, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17162982339
Failed job 17162261239, check-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/17162261239
Failed job 17162261192, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/17162261192
failed pipeline 2929887766, total jobs 120, skipped 7, failed 18, 0 tests, 0 skipped tests, 0 failed tests
Failed job 17055360127, check-python-minreqs, https://gitlab.com/qemu-project/qemu/-/jobs/17055360127
Failed job 17055360119, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/17055360119
Failed job 17055360117, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/17055360117
Failed job 17055360116, check-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/17055360116
Failed job 17055360113, block: [vpc], https://gitlab.com/qemu-project/qemu/-/jobs/17055360113
Failed job 17055360112, block: [vmdk], https://gitlab.com/qemu-project/qemu/-/jobs/17055360112
Failed job 17055360111, block: [vhdx], https://gitlab.com/qemu-project/qemu/-/jobs/17055360111
Failed job 17055360110, block: [vdi], https://gitlab.com/qemu-project/qemu/-/jobs/17055360110
Failed job 17055360109, block: [raw], https://gitlab.com/qemu-project/qemu/-/jobs/17055360109
Failed job 17055360108, block: [qed], https://gitlab.com/qemu-project/qemu/-/jobs/17055360108
Failed job 17055360107, block: [qcow2], https://gitlab.com/qemu-project/qemu/-/jobs/17055360107
Failed job 17055360106, block: [parallels], https://gitlab.com/qemu-project/qemu/-/jobs/17055360106
Failed job 17055360105, block: [nbd], https://gitlab.com/qemu-project/qemu/-/jobs/17055360105
Failed job 17055360104, block: [luks], https://gitlab.com/qemu-project/qemu/-/jobs/17055360104
Failed job 17055360078, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/17055360078
Failed job 17055360066, tsan-build, https://gitlab.com/qemu-project/qemu/-/jobs/17055360066
Failed job 17055360065, build-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/17055360065
Failed job 17055360061, clang-system, https://gitlab.com/qemu-project/qemu/-/jobs/17055360061
failed pipeline 2908947622, total jobs 119, skipped 5, failed 5, 0 tests, 0 skipped tests, 0 failed tests
Failed job 16911683051, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16911683051
Failed job 16911681150, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16911681150
Failed job 16911679766, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16911679766
Failed job 16911678710, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16911678710
Failed job 16911433872, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16911433872
failed pipeline 2906503036, total jobs 119, skipped 5, failed 4, 0 tests, 0 skipped tests, 0 failed tests
Failed job 16896095104, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16896095104
Failed job 16896090260, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16896090260
Failed job 16896083592, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16896083592
Failed job 16895432496, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16895432496
failed pipeline 2876539219, total jobs 119, skipped 5, failed 6, 0 tests, 0 skipped tests, 0 failed tests
Failed job 16692899473, functional-system-fedora, https://gitlab.com/qemu-project/qemu/-/jobs/16692899473
Failed job 16692899464, aarch64-macos-26-build, https://gitlab.com/qemu-project/qemu/-/jobs/16692899464
Failed job 16692899463, aarch64-macos-15-build, https://gitlab.com/qemu-project/qemu/-/jobs/16692899463
Failed job 16692899432, clang-system, https://gitlab.com/qemu-project/qemu/-/jobs/16692899432
Failed job 16692899429, build-some-softmmu, https://gitlab.com/qemu-project/qemu/-/jobs/16692899429
Failed job 16692899413, cross-win64-system, https://gitlab.com/qemu-project/qemu/-/jobs/16692899413
failed pipeline 2840522620, total jobs 119, skipped 5, failed 7, 0 tests, 0 skipped tests, 0 failed tests
Failed job 16454091776, functional-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16454091776
Failed job 16454091774, functional-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16454091774
Failed job 16454091761, check-cfi-x86_64, https://gitlab.com/qemu-project/qemu/-/jobs/16454091761
Failed job 16454091760, check-system-opensuse, https://gitlab.com/qemu-project/qemu/-/jobs/16454091760
Failed job 16454091758, functional-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16454091758
Failed job 16454091740, check-system-centos, https://gitlab.com/qemu-project/qemu/-/jobs/16454091740
Failed job 16454091730, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16454091730
failed pipeline 2839257817, total jobs 119, skipped 5, failed 2, 0 tests, 0 skipped tests, 0 failed tests
Failed job 16436167533, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16436167533
Failed job 16436167532, ubuntu-24.04-s390x-all-linux, https://gitlab.com/qemu-project/qemu/-/jobs/16436167532
failed pipeline 2830538304, total jobs 119, skipped 5, failed 3, 0 tests, 0 skipped tests, 0 failed tests
Failed job 16377263801, check-rust-tools-nightly, https://gitlab.com/qemu-project/qemu/-/jobs/16377263801
Failed job 16377263686, ubuntu-24.04-s390x-all-system, https://gitlab.com/qemu-project/qemu/-/jobs/16377263686
Failed job 16377263684, ubuntu-24.04-s390x-all-linux, https://gitlab.com/qemu-project/qemu/-/jobs/16377263684
failed pipeline 2820341086, total jobs 119, skipped 5, failed 2, 0 tests, 0 skipped tests, 0 failed tests
Failed job 16307778927, check-rust-tools-nightly, https://gitlab.com/qemu-project/qemu/-/jobs/16307778927
Failed job 16307778897, functional-system-debian, https://gitlab.com/qemu-project/qemu/-/jobs/16307778897
It has rudimentary access to the tests but I think the deep dive would
require better retention of the job artefacts. The second pipeline in
the above list lists the "job artefacts" as removed 3hrs ago which is I
guess why we don't get the test breakdown for the earlier jobs.
Anyway perhaps we can expand this to be more useful?
>
> thanks
> -- PMM
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
next prev parent reply other threads:[~2026-10-21 10:57 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-20 23:16 [PATCH] ci: add automatic retry Pierrick Bouvier
2026-10-21 2:41 ` Brian Cain
2026-10-21 5:09 ` Pierrick Bouvier
2026-10-21 9:20 ` Peter Maydell
2026-10-21 10:56 ` Alex Bennée [this message]
2026-10-21 15:59 ` Pierrick Bouvier
2026-10-21 8:16 ` Richard Henderson
2026-10-22 17:41 ` Pierrick Bouvier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87fqxz1fv7.fsf@draig.linaro.org \
--to=alex.bennee@linaro.org \
--cc=berrange@redhat.com \
--cc=brian.cain@oss.qualcomm.com \
--cc=pbonzini@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=philmd@oss.qualcomm.com \
--cc=pierrick.bouvier@oss.qualcomm.com \
--cc=qemu-devel@nongnu.org \
--cc=richard.henderson@linaro.org \
--cc=stefanha@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.