From: "Daniel P. Berrangé" <berrange@redhat.com>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>, qemu-devel@nongnu.org
Subject: Re: [PULL 00/11] QEMU changes for 2021-03-02
Date: Fri, 4 Mar 2022 19:15:22 +0000 [thread overview]
Message-ID: <YiJlSlJube4dOk/m@redhat.com> (raw)
In-Reply-To: <CAFEAcA--dtmffH4FJUuuE1d6yR-4Mweu481p_y-EsJKEtPRjTw@mail.gmail.com>
On Fri, Mar 04, 2022 at 06:46:51PM +0000, Peter Maydell wrote:
> On Fri, 4 Mar 2022 at 17:41, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > The test seems to be flaky, I've been fighting with it all week---trying
> > multiple versions of this pull request and removing patches until
> > build-oss-fuzz passed. The set of patches that triggered it or not was
> > completely random, but I'll not that it did pass with this exact commit
> > I'm submitting (https://gitlab.com/bonzini/qemu/-/jobs/2154365356).
> >
> > I wanted to look at this today again before replying to you, but as you
> > know I was sidetracked by work on the qemu.org infrastructure. So, I
> > can look at this but I really need to ask you one of two favors:
> >
> > 1) decide that the test is flaky and merge this pull request, and then
> > I'll send before Monday the changes that I've omitted here (which again
> > have nothing to do with qos-test). I'll look at qos-test during soft
> > freeze.
> >
> > 2) accept that I'll send another x86 pull request (not a large one)
> > after soft freeze, so that I have more time to debug this (likely
> > unrelated) build-oss-fuzz issue.
>
> Either of these is fine; my requirement is only that either:
> (1) the oss-fuzz gitlab CI job needs to in practice actually
> pass at least most of the time
> (2) we need to switch it to ok-to-fail or disable it
>
> so I don't have CI failing for every merge I make.
This is far from the first time that oss-fuzz has caused us pain. It
feels like it has been flaky for prolonged periods of time, for as
long as it has existed.
When I tried to switch CI to use Fedora 35 oss-fuzz was consistently
failing for months for no obvious reason that I could determine
despite days of debugging. Then one day I woke up and it magically
started working again, for no obvious reason. Inexplicable.
Conceptually we benefit from fuzzing to find obscure bugs.
Have we actually found any real bugs from the oss-fuzz CI
job we have though ? Between us all, we've certainly lost
many many many days of developer time debugging the thing
for false failures.
The magic set of args needed to run it make it even more
unpleasant to deal with, and that scripts/oss-fuzz/build.sh
runs different code inside GitLab vs outside GitLab also
make debugging worse.
Personally I favour turning it into a non-gating job as
I don't want to invest more of my own time debugging
non-bugs in it.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
next prev parent reply other threads:[~2022-03-04 19:22 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-02 18:11 [PULL 00/11] QEMU changes for 2021-03-02 Paolo Bonzini
2022-03-02 18:11 ` [PULL 01/11] whpx: Fixed reporting of the CPU context to GDB for 64-bit Paolo Bonzini
2022-03-02 18:11 ` [PULL 02/11] whpx: Fixed incorrect CR8/TPR synchronization Paolo Bonzini
2022-03-02 18:11 ` [PULL 03/11] vmxcap: Add 5-level EPT bit Paolo Bonzini
2022-03-02 18:11 ` [PULL 04/11] meson: fix generic location of vss headers Paolo Bonzini
2022-03-02 18:11 ` [PULL 05/11] qga/vss-win32: check old VSS SDK headers Paolo Bonzini
2022-03-02 18:11 ` [PULL 06/11] qga/vss: update informative message about MinGW Paolo Bonzini
2022-03-02 18:11 ` [PULL 07/11] update meson-buildoptions.sh Paolo Bonzini
2022-03-02 18:11 ` [PULL 08/11] kvm-irqchip: introduce new API to support route change Paolo Bonzini
2022-03-02 18:11 ` [PULL 09/11] kvm/msi: do explicit commit when adding msi routes Paolo Bonzini
2022-03-02 18:11 ` [PULL 10/11] target/i386: only include bits in pg_mode if they are not ignored Paolo Bonzini
2022-03-02 18:11 ` [PULL 11/11] target/i386: Throw a #SS when loading a non-canonical IST Paolo Bonzini
2022-03-02 20:55 ` [PULL 00/11] QEMU changes for 2021-03-02 Peter Maydell
2022-03-04 17:41 ` Paolo Bonzini
2022-03-04 18:46 ` Peter Maydell
2022-03-04 19:15 ` Daniel P. Berrangé [this message]
2022-03-04 19:22 ` Peter Maydell
2022-03-04 19:30 ` Daniel P. Berrangé
2022-03-04 21:20 ` Paolo Bonzini
2022-03-04 22:32 ` Richard Henderson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YiJlSlJube4dOk/m@redhat.com \
--to=berrange@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).