Re: [RFC PATCH] scripts/ci: add gitlab-failure-analysis script

["Daniel P. Berrangé" <berrange@redhat.com>]

On Tue, Sep 09, 2025 at 10:00:05AM +0100, Peter Maydell wrote:
> On Tue, 9 Sept 2025 at 09:39, Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> > Thomas Huth <thuth@redhat.com> writes:
> >
> > > On 08/09/2025 23.18, Alex Bennée wrote:
> > >> This is a script designed to collect data from multiple pipelines and
> > >> analyse the failure modes they have.
> > >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > >> ---
> > >>   scripts/ci/gitlab-failure-analysis | 65 ++++++++++++++++++++++++++++++
> > >>   1 file changed, 65 insertions(+)
> > >>   create mode 100755 scripts/ci/gitlab-failure-analysis
> > >
> > > You already get a nice overview by visiting a page like
> > > https://gitlab.com/qemu-project/qemu/-/pipelines/2019002986 ... what's
> > > the advantage of this script?
> >
> > Not having to click every link when I want to see what the pattern of
> > failures is and what might be a candidate for making flaky.
> 
> What I would like for finding flaky tests is to find every
> case where:
>  * a job failed on commit hash X
>  * we also have the same job succeeding on the same commit X
> 
> Those are the flaky tests, where we hit retry and it just
> passed the second time, and it rules out the cases where
> we had a genuine "job failed because the code being tested
> for merge had a problem".
> 
> When we find those jobs that only failed because of a flaky
> test then we can analyse their logs to identify what the
> exact failures were.
> 
> Can we find those with this script ?  (You can't do it with
> the gitlab web UI, whose search and filter capabilities
> are extremely limited.)

Downloading data from gitlab API is painfully slow so not something
you want to do regularly/repeatedly.

If we can have the script to download the data and save it locally,
we could then do something like populate a sqllite DB with pipeline
results which can we efficiently query to extract failure patterns.
I guess this script at least starts us moving in that direction by
giving us the framework to fetch data, and build on that...

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|