[PATCH] tests/qtest: Raise the ide-test timeout

[PATCH] tests/qtest: Raise the ide-test timeout

[Peter Maydell]

The ide-test occasionally times out: on the system I run
vm-build-openbsd on, it usually takes about 18 seconds, but
occasionally hits the 60s timeout, likely when the host machine is
under heavy load.  I have also seen this test hit its time limit on
the s390x CI runner.

Double the timeout for this test so that it won't hit its timeout
even when the host is running more slowly than usual.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 tests/qtest/meson.build | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index b207e386965..e8be8b3942d 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -4,6 +4,7 @@ slow_qtests = {
   'bios-tables-test' : 910,
   'cdrom-test' : 610,
   'device-introspect-test' : 720,
+  'ide-test' : 120,
   'migration-test' : 480,
   'npcm7xx_pwm-test': 300,
   'npcm7xx_watchdog_timer-test': 120,
-- 
2.34.1

Re: [PATCH] tests/qtest: Raise the ide-test timeout

[Thomas Huth]

On 15/10/2024 13.37, Peter Maydell wrote:
> The ide-test occasionally times out: on the system I run
> vm-build-openbsd on, it usually takes about 18 seconds, but
> occasionally hits the 60s timeout, likely when the host machine is
> under heavy load.  I have also seen this test hit its time limit on
> the s390x CI runner.
> 
> Double the timeout for this test so that it won't hit its timeout
> even when the host is running more slowly than usual.
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
>   tests/qtest/meson.build | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
> index b207e386965..e8be8b3942d 100644
> --- a/tests/qtest/meson.build
> +++ b/tests/qtest/meson.build
> @@ -4,6 +4,7 @@ slow_qtests = {
>     'bios-tables-test' : 910,
>     'cdrom-test' : 610,
>     'device-introspect-test' : 720,
> +  'ide-test' : 120,
>     'migration-test' : 480,
>     'npcm7xx_pwm-test': 300,
>     'npcm7xx_watchdog_timer-test': 120,

Reviewed-by: Thomas Huth <thuth@redhat.com>

PS: It's a bit weird that we see more and more timeout problems recently ... 
I think I've also seen some local ones on my laptop when my system was under 
load, but not really reproducible. I wonder whether we have a performance 
degradation somewhere in the code...?

Re: [PATCH] tests/qtest: Raise the ide-test timeout

[Peter Maydell]

On Tue, 15 Oct 2024 at 12:42, Thomas Huth <thuth@redhat.com> wrote:
>
> On 15/10/2024 13.37, Peter Maydell wrote:
> > The ide-test occasionally times out: on the system I run
> > vm-build-openbsd on, it usually takes about 18 seconds, but
> > occasionally hits the 60s timeout, likely when the host machine is
> > under heavy load.  I have also seen this test hit its time limit on
> > the s390x CI runner.
> >
> > Double the timeout for this test so that it won't hit its timeout
> > even when the host is running more slowly than usual.
> >
> > Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> > ---
> >   tests/qtest/meson.build | 1 +
> >   1 file changed, 1 insertion(+)
> >
> > diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
> > index b207e386965..e8be8b3942d 100644
> > --- a/tests/qtest/meson.build
> > +++ b/tests/qtest/meson.build
> > @@ -4,6 +4,7 @@ slow_qtests = {
> >     'bios-tables-test' : 910,
> >     'cdrom-test' : 610,
> >     'device-introspect-test' : 720,
> > +  'ide-test' : 120,
> >     'migration-test' : 480,
> >     'npcm7xx_pwm-test': 300,
> >     'npcm7xx_watchdog_timer-test': 120,
>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
>
> PS: It's a bit weird that we see more and more timeout problems recently ...
> I think I've also seen some local ones on my laptop when my system was under
> load, but not really reproducible. I wonder whether we have a performance
> degradation somewhere in the code...?

If it is, it's one where it's intermittent whether it hits or not
(e.g. I grepped a bunch of my local logs for this test and there's
a lot of 18s and a 30s and a couple of 60s) and it tends to hit for
multiple tests in a single CI run when it does hit. e.g.

https://gitlab.com/qemu-project/qemu/-/jobs/8065512834

https://gitlab.com/qemu-project/qemu/-/jobs/8069311713

are a failure and a pass of the same git hash for "clang-system".
The 'make check' stage on the failure took just under 40 minutes;
the same 'make check' on the passing job took less than 3 !
The 'make' stage also had a similar gap in timing between the
two, and I don't think that's entirely because of ccache caching
effects because the stats at the bottom of the passing job
say the hitrate was >90%.

-- PMM

Re: [PATCH] tests/qtest: Raise the ide-test timeout

[Peter Maydell]

On Tue, 15 Oct 2024 at 13:01, Peter Maydell <peter.maydell@linaro.org> wrote:
> If it is, it's one where it's intermittent whether it hits or not
> (e.g. I grepped a bunch of my local logs for this test and there's
> a lot of 18s and a 30s and a couple of 60s) and it tends to hit for
> multiple tests in a single CI run when it does hit. e.g.
>
> https://gitlab.com/qemu-project/qemu/-/jobs/8065512834
>
> https://gitlab.com/qemu-project/qemu/-/jobs/8069311713
>
> are a failure and a pass of the same git hash for "clang-system".
> The 'make check' stage on the failure took just under 40 minutes;
> the same 'make check' on the passing job took less than 3 !
> The 'make' stage also had a similar gap in timing between the
> two, and I don't think that's entirely because of ccache caching
> effects because the stats at the bottom of the passing job
> say the hitrate was >90%.

I managed to both misread the ccache stats and then miswrite
what I intended to say about them. Rereading them, what
they say is that the hitrate is less than 1%...

-- PMM

Re: [PATCH] tests/qtest: Raise the ide-test timeout

[Peter Maydell]

On Tue, 15 Oct 2024 at 12:37, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> The ide-test occasionally times out: on the system I run
> vm-build-openbsd on, it usually takes about 18 seconds, but
> occasionally hits the 60s timeout, likely when the host machine is
> under heavy load.  I have also seen this test hit its time limit on
> the s390x CI runner.
>
> Double the timeout for this test so that it won't hit its timeout
> even when the host is running more slowly than usual.
>
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---

I'll put this into my target-arm.next queue unless
you'd prefer it to go in some other way.

thanks
-- PMM