From: "Daniel P. Berrangé" <berrange@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Cleber Rosa" <crosa@redhat.com>,
"Thomas Huth" <thuth@redhat.com>, "John Snow" <jsnow@redhat.com>,
"Paolo Bonzini" <pbonzini@redhat.com>,
"Laurent Vivier" <lvivier@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>,
"Daniel P. Berrangé" <berrange@redhat.com>
Subject: [PATCH v2 12/12] mtest2make: stop disabling meson test timeouts
Date: Mon, 17 Jul 2023 19:28:58 +0100 [thread overview]
Message-ID: <20230717182859.707658-13-berrange@redhat.com> (raw)
In-Reply-To: <20230717182859.707658-1-berrange@redhat.com>
The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab CI and a test gets stuck. It will stall until GitLab kills the
CI job. This leaves us with little easily consumable information about
the stalled test. The TAP format doesn't show the test name until it is
completed, and TAP output from multiple tests it interleaved. So we
have to analyse the log to figure out what tests had un-finished TAP
output present and thus infer which test case caused the hang. This is
very time consuming and error prone.
By allowing meson to kill stalled tests, we get a direct display of what
test program got stuck, which lets us more directly focus in on what
specific test case within the test program hung.
The other issue with disabling meson test timeouts by default is that it
makes it more likely that maintainers inadvertantly introduce slowdowns.
For example the recent-ish change that accidentally made migrate-test
take 15-20 minutes instead of around 1 minute.
The main risk of this change is that the individual test timeouts might
be too short to allow completion in high load scenarios. Thus, there is
likely to be some short term pain where we have to bump the timeouts for
certain tests to make them reliable enough. The preceeding few patches
raised the timeouts for all failures that were immediately apparent
in GitLab CI.
Even with the possible short term instability, this should still be a
net win for debuggability of failed CI pipelines over the long term.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
scripts/mtest2make.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 179dd54871..eb01a05ddb 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -27,7 +27,8 @@ def names(self, base):
.speed.slow = $(foreach s,$(sort $(filter-out %-thorough, $1)), --suite $s)
.speed.thorough = $(foreach s,$(sort $1), --suite $s)
-.mtestargs = --no-rebuild -t 0
+TIMEOUT_MULTIPLIER = 1
+.mtestargs = --no-rebuild -t $(TIMEOUT_MULTIPLIER)
ifneq ($(SPEED), quick)
.mtestargs += --setup $(SPEED)
endif
--
2.41.0
next prev parent reply other threads:[~2023-07-17 18:29 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-17 18:28 [PATCH v2 00/12] tests: enable meson test timeouts to improve debuggability Daniel P. Berrangé
2023-07-17 18:28 ` [PATCH v2 01/12] qtest: bump min meson timeout to 60 seconds Daniel P. Berrangé
2023-07-17 18:28 ` [PATCH v2 02/12] qtest: bump migration-test timeout to 5 minutes Daniel P. Berrangé
2023-07-17 18:28 ` [PATCH v2 03/12] qtest: bump qom-test timeout to 15 minutes Daniel P. Berrangé
2023-07-17 18:28 ` [PATCH v2 04/12] qtest: bump npcm7xx_pwn-test timeout to 5 minutes Daniel P. Berrangé
2023-07-19 13:06 ` Thomas Huth
2023-07-19 13:38 ` Peter Maydell
2023-07-17 18:28 ` [PATCH v2 05/12] qtest: bump test-hmp timeout to 4 minutes Daniel P. Berrangé
2023-07-19 13:07 ` Thomas Huth
2023-07-17 18:28 ` [PATCH v2 06/12] qtest: bump pxe-test timeout to 3 minutes Daniel P. Berrangé
2023-07-19 13:08 ` Thomas Huth
2023-07-19 13:26 ` Thomas Huth
2023-09-12 15:40 ` Daniel P. Berrangé
2023-07-17 18:28 ` [PATCH v2 07/12] qtest: bump prom-env-test " Daniel P. Berrangé
2023-07-19 13:22 ` Thomas Huth
2023-07-19 13:37 ` Thomas Huth
2023-07-19 13:38 ` Daniel P. Berrangé
2023-07-17 18:28 ` [PATCH v2 08/12] qtest: bump boot-serial-test " Daniel P. Berrangé
2023-07-19 13:24 ` Thomas Huth
2023-07-17 18:28 ` [PATCH v2 09/12] qtest: bump qos-test timeout to 2 minutes Daniel P. Berrangé
2023-07-19 13:28 ` Thomas Huth
2023-07-17 18:28 ` [PATCH v2 10/12] qtest: bump aspeed_smc-test timeout to 4 minutes Daniel P. Berrangé
2023-07-19 13:35 ` Thomas Huth
2023-07-17 18:28 ` [PATCH v2 11/12] qtest: bump bios-table-test timeout to 9 minutes Daniel P. Berrangé
2023-07-17 18:28 ` Daniel P. Berrangé [this message]
2023-07-18 6:29 ` [PATCH v2 12/12] mtest2make: stop disabling meson test timeouts Philippe Mathieu-Daudé
2023-08-08 8:57 ` [PATCH v2 00/12] tests: enable meson test timeouts to improve debuggability Alex Bennée
2023-08-13 7:02 ` Thomas Huth
2023-08-17 10:36 ` Daniel P. Berrangé
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230717182859.707658-13-berrange@redhat.com \
--to=berrange@redhat.com \
--cc=alex.bennee@linaro.org \
--cc=crosa@redhat.com \
--cc=jsnow@redhat.com \
--cc=lvivier@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=thuth@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).