* [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes
@ 2023-09-12 13:38 Stefan Hajnoczi
2023-09-12 13:52 ` Daniel P. Berrangé
0 siblings, 1 reply; 6+ messages in thread
From: Stefan Hajnoczi @ 2023-09-12 13:38 UTC (permalink / raw)
To: qemu-devel
Cc: Daniel Berrange, Thomas Huth, Beraldo Leal, Alex Bennée,
Philippe Mathieu-Daudé, Wainer dos Santos Moschetta,
Stefan Hajnoczi
The 80m timeout is not enough:
672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed
Timed out!
https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882
Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes")
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
.gitlab-ci.d/cirrus.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 41d64d6680..d19633f758 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -15,7 +15,7 @@
stage: build
image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
needs: []
- timeout: 80m
+ timeout: 100m
allow_failure: true
script:
- source .gitlab-ci.d/cirrus/$NAME.vars
--
2.41.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes
2023-09-12 13:38 [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes Stefan Hajnoczi
@ 2023-09-12 13:52 ` Daniel P. Berrangé
2023-09-12 14:02 ` Stefan Hajnoczi
0 siblings, 1 reply; 6+ messages in thread
From: Daniel P. Berrangé @ 2023-09-12 13:52 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: qemu-devel, Thomas Huth, Beraldo Leal, Alex Bennée,
Philippe Mathieu-Daudé, Wainer dos Santos Moschetta
On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> The 80m timeout is not enough:
>
> 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed
> Timed out!
IIUC, that 'timed out' message is coming from Cirrus CI logs, which
we can see over on the cirrus task:
https://cirrus-ci.com/task/6462328380588032
> https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
IOW, we're not hitting the gitlab timeout, we're hitting hte
Cirrus CI timeout, which defaults to 60 minutes. The other
4 minuts gitlab reports is likely because Cirrus queued the
job for 4 minutes before starting execution.
>
> Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882
> Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes")
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> .gitlab-ci.d/cirrus.yml | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> index 41d64d6680..d19633f758 100644
> --- a/.gitlab-ci.d/cirrus.yml
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -15,7 +15,7 @@
> stage: build
> image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> needs: []
> - timeout: 80m
> + timeout: 100m
> allow_failure: true
> script:
> - source .gitlab-ci.d/cirrus/$NAME.vars
IIUC, we need to put a 'timeout_in' setting someone in
.gitlab-ci.d/cirrus/build.yml instead, to override
Cirrus 60 minute limit:
https://cirrus-ci.org/faq/#instance-timed-out
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes
2023-09-12 13:52 ` Daniel P. Berrangé
@ 2023-09-12 14:02 ` Stefan Hajnoczi
2023-09-12 14:15 ` Daniel P. Berrangé
0 siblings, 1 reply; 6+ messages in thread
From: Stefan Hajnoczi @ 2023-09-12 14:02 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Stefan Hajnoczi, qemu-devel, Thomas Huth, Beraldo Leal,
Alex Bennée, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta
On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > The 80m timeout is not enough:
> >
> > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed
> > Timed out!
>
> IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> we can see over on the cirrus task:
>
> https://cirrus-ci.com/task/6462328380588032
>
> > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
>
> This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
>
> IOW, we're not hitting the gitlab timeout, we're hitting hte
> Cirrus CI timeout, which defaults to 60 minutes. The other
> 4 minuts gitlab reports is likely because Cirrus queued the
> job for 4 minutes before starting execution.
I'm glad you spotted that. I'm not familiar with Cirrus. Could you
send a patch that sets 'timeout_in'?
Thanks,
Stefan
>
> >
> > Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882
> > Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes")
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> > .gitlab-ci.d/cirrus.yml | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> > index 41d64d6680..d19633f758 100644
> > --- a/.gitlab-ci.d/cirrus.yml
> > +++ b/.gitlab-ci.d/cirrus.yml
> > @@ -15,7 +15,7 @@
> > stage: build
> > image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> > needs: []
> > - timeout: 80m
> > + timeout: 100m
> > allow_failure: true
> > script:
> > - source .gitlab-ci.d/cirrus/$NAME.vars
>
> IIUC, we need to put a 'timeout_in' setting someone in
> .gitlab-ci.d/cirrus/build.yml instead, to override
> Cirrus 60 minute limit:
>
> https://cirrus-ci.org/faq/#instance-timed-out
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes
2023-09-12 14:02 ` Stefan Hajnoczi
@ 2023-09-12 14:15 ` Daniel P. Berrangé
2023-09-12 14:17 ` Stefan Hajnoczi
2023-09-12 16:36 ` Daniel P. Berrangé
0 siblings, 2 replies; 6+ messages in thread
From: Daniel P. Berrangé @ 2023-09-12 14:15 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Stefan Hajnoczi, qemu-devel, Thomas Huth, Beraldo Leal,
Alex Bennée, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta
On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote:
> On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > > The 80m timeout is not enough:
> > >
> > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed
> > > Timed out!
> >
> > IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> > we can see over on the cirrus task:
> >
> > https://cirrus-ci.com/task/6462328380588032
> >
> > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
> >
> > This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
> >
> > IOW, we're not hitting the gitlab timeout, we're hitting hte
> > Cirrus CI timeout, which defaults to 60 minutes. The other
> > 4 minuts gitlab reports is likely because Cirrus queued the
> > job for 4 minutes before starting execution.
>
> I'm glad you spotted that. I'm not familiar with Cirrus. Could you
> send a patch that sets 'timeout_in'?
Yes, testing now
https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434
https://gitlab.com/berrange/qemu/-/jobs/5069195895
https://cirrus-ci.com/task/5135339078025216
The cirrus CI job page looks to be picking up the elevated timeout.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes
2023-09-12 14:15 ` Daniel P. Berrangé
@ 2023-09-12 14:17 ` Stefan Hajnoczi
2023-09-12 16:36 ` Daniel P. Berrangé
1 sibling, 0 replies; 6+ messages in thread
From: Stefan Hajnoczi @ 2023-09-12 14:17 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Stefan Hajnoczi, qemu-devel, Thomas Huth, Beraldo Leal,
Alex Bennée, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta
Thank you!
Stefan
On Tue, 12 Sept 2023 at 10:15, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote:
> > On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > > > The 80m timeout is not enough:
> > > >
> > > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed
> > > > Timed out!
> > >
> > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> > > we can see over on the cirrus task:
> > >
> > > https://cirrus-ci.com/task/6462328380588032
> > >
> > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
> > >
> > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
> > >
> > > IOW, we're not hitting the gitlab timeout, we're hitting hte
> > > Cirrus CI timeout, which defaults to 60 minutes. The other
> > > 4 minuts gitlab reports is likely because Cirrus queued the
> > > job for 4 minutes before starting execution.
> >
> > I'm glad you spotted that. I'm not familiar with Cirrus. Could you
> > send a patch that sets 'timeout_in'?
>
> Yes, testing now
>
> https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434
>
> https://gitlab.com/berrange/qemu/-/jobs/5069195895
>
> https://cirrus-ci.com/task/5135339078025216
>
> The cirrus CI job page looks to be picking up the elevated timeout.
>
>
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes
2023-09-12 14:15 ` Daniel P. Berrangé
2023-09-12 14:17 ` Stefan Hajnoczi
@ 2023-09-12 16:36 ` Daniel P. Berrangé
1 sibling, 0 replies; 6+ messages in thread
From: Daniel P. Berrangé @ 2023-09-12 16:36 UTC (permalink / raw)
To: Stefan Hajnoczi, Stefan Hajnoczi, qemu-devel, Thomas Huth,
Beraldo Leal, Alex Bennée, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta
On Tue, Sep 12, 2023 at 03:15:28PM +0100, Daniel P. Berrangé wrote:
> On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote:
> > On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote:
> > > > The 80m timeout is not enough:
> > > >
> > > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed
> > > > Timed out!
> > >
> > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which
> > > we can see over on the cirrus task:
> > >
> > > https://cirrus-ci.com/task/6462328380588032
> > >
> > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599
> > >
> > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20.
> > >
> > > IOW, we're not hitting the gitlab timeout, we're hitting hte
> > > Cirrus CI timeout, which defaults to 60 minutes. The other
> > > 4 minuts gitlab reports is likely because Cirrus queued the
> > > job for 4 minutes before starting execution.
> >
> > I'm glad you spotted that. I'm not familiar with Cirrus. Could you
> > send a patch that sets 'timeout_in'?
>
> Yes, testing now
>
> https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434
>
> https://gitlab.com/berrange/qemu/-/jobs/5069195895
>
> https://cirrus-ci.com/task/5135339078025216
>
> The cirrus CI job page looks to be picking up the elevated timeout.
Still fails, and given it previously worked at 35 minutes, needing
a 1h30 timeout would be incredulous. This appears to be a genuine
hang in QEMU's test suite on FreeBSD 13.2.
I'm spinning up a VM to debug further
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-09-12 16:37 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-09-12 13:38 [PATCH] gitlab-ci/cirrus: Increase timeout to 100 minutes Stefan Hajnoczi
2023-09-12 13:52 ` Daniel P. Berrangé
2023-09-12 14:02 ` Stefan Hajnoczi
2023-09-12 14:15 ` Daniel P. Berrangé
2023-09-12 14:17 ` Stefan Hajnoczi
2023-09-12 16:36 ` Daniel P. Berrangé
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).