From: Thomas Huth <thuth@redhat.com>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: qemu-devel@nongnu.org, qemu-arm@nongnu.org,
"Philippe Mathieu-Daudé" <philmd@linaro.org>,
"Marcin Juszkiewicz" <marcin.juszkiewicz@linaro.org>,
"Leif Lindholm" <quic_llindhol@quicinc.com>,
"Radoslaw Biernacki" <rad@semihalf.com>,
"Beraldo Leal" <bleal@redhat.com>,
"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
"Cleber Rosa" <crosa@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>
Subject: Re: [PATCH] tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI
Date: Mon, 10 Jul 2023 18:44:23 +0200 [thread overview]
Message-ID: <0c70e8f3-586f-5683-7ef2-0aa3138d7832@redhat.com> (raw)
In-Reply-To: <CAFEAcA_QTRJwtQxh1FRHtwjSARAptF-j1ZeyGen8xBfrhq7tww@mail.gmail.com>
On 10/07/2023 16.54, Peter Maydell wrote:
> On Mon, 10 Jul 2023 at 15:43, Thomas Huth <thuth@redhat.com> wrote:
>>
>> The test fails occasionally, see e.g.:
>>
>> https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
>> https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290
>>
>> Disable it at least in the gitlab-CI to avoid failing CI
>> pipelines due to this problem.
>
> I'm not really a fan of disabling flaky tests only in the
> gitlab CI. It means that plain "make check-avocado"
> is useless to the average developer because it can
> have all manner of broken or flaky tests in it that
> never get tested by our CI. In fact I've had to develop
> the habit of running "GITLAB_CI=1 make check-avocado"
> so I can run the CI tests that we think work and not
> get distracted by intermittent failures.
>
> If we want "disable this test except for if you're
> trying to investigate why it's flaky" then we should
> have the test be opt-in, not opt-out, the same way
> we do with QEMU_TEST_FLAKY_TESTS in the qtests.
Agreed, this makes perfectly sense. Actually, I was looking for something
like this in the tests/avocado directory, but I did not find anything
similar, so I went for the GITLAB_CI variable ...
I'll send a v2 with QEMU_TEST_FLAKY_TESTS instead (and document it in the
docs directory).
Thomas
prev parent reply other threads:[~2023-07-10 16:45 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-10 14:43 [PATCH] tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI Thomas Huth
2023-07-10 14:54 ` Peter Maydell
2023-07-10 16:44 ` Thomas Huth [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0c70e8f3-586f-5683-7ef2-0aa3138d7832@redhat.com \
--to=thuth@redhat.com \
--cc=alex.bennee@linaro.org \
--cc=bleal@redhat.com \
--cc=crosa@redhat.com \
--cc=marcin.juszkiewicz@linaro.org \
--cc=peter.maydell@linaro.org \
--cc=philmd@linaro.org \
--cc=qemu-arm@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=quic_llindhol@quicinc.com \
--cc=rad@semihalf.com \
--cc=wainersm@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).