[PATCH RFC 0/2] dynamic timeout for tests

[PATCH RFC 0/2] dynamic timeout for tests

[Michael S. Tsirkin]

so we keep making timeouts longer for CI but one has to stop
somewhere. netdev socket test recently failed for me again
even though it's at 2 minutes already.
here's an experiment for netdev-socket.
if this works well for a while we can generalize to other
tests.

Michael S. Tsirkin (2):
  osdep: add getloadavg
  netdev: set timeout depending on loadavg

 include/qemu/osdep.h        | 10 ++++++++++
 tests/qtest/netdev-socket.c | 28 +++++++++++++++++++++++++++-
 meson.build                 |  1 +
 3 files changed, 38 insertions(+), 1 deletion(-)

-- 
MST

[PATCH RFC 1/2] osdep: add getloadavg

[Michael S. Tsirkin]

getloadavg is supported on Linux, BSDs, Solaris.

Following man page:
RETURN VALUE
       If the load average was unobtainable, -1 is returned; otherwise,
       the number of samples actually retrieved is returned.

accordingly, make stub for systems which don't support this function return -1
for consistency.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 include/qemu/osdep.h | 10 ++++++++++
 meson.build          |  1 +
 2 files changed, 11 insertions(+)

diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index 475a1c62ff..d30ba73eda 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -779,6 +779,16 @@ static inline int platform_does_not_support_system(const char *command)
 }
 #endif /* !HAVE_SYSTEM_FUNCTION */
 
+/**
+ * If the load average was unobtainable, -1 is returned
+ */
+#ifndef HAVE_GETLOADAVG_FUNCTION
+static inline int getloadavg(double loadavg[], int nelem)
+{
+    return -1;
+}
+#endif /* !HAVE_GETLOADAVG_FUNCTION */
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/meson.build b/meson.build
index dcef8b1e79..7f333cb80e 100644
--- a/meson.build
+++ b/meson.build
@@ -2254,6 +2254,7 @@ config_host_data.set('HAVE_GLIB_WITH_SLICE_ALLOCATOR', glib_has_gslice)
 config_host_data.set('HAVE_OPENPTY', cc.has_function('openpty', dependencies: util))
 config_host_data.set('HAVE_STRCHRNUL', cc.has_function('strchrnul'))
 config_host_data.set('HAVE_SYSTEM_FUNCTION', cc.has_function('system', prefix: '#include <stdlib.h>'))
+config_host_data.set('HAVE_GETLOADAVG_FUNCTION', cc.has_function('getloadavg', prefix: '#include <stdlib.h>'))
 if rbd.found()
   config_host_data.set('HAVE_RBD_NAMESPACE_EXISTS',
                        cc.has_function('rbd_namespace_exists',
-- 
MST

[PATCH RFC 2/2] netdev: set timeout depending on loadavg

[Michael S. Tsirkin]

netdev test keeps failing sometimes.
I don't think we should increase the timeout some more:
let's try something else instead, testing how busy the
system is.

Seems to work for me.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 tests/qtest/netdev-socket.c | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/tests/qtest/netdev-socket.c b/tests/qtest/netdev-socket.c
index 7ba1eff120..bb99d08b5e 100644
--- a/tests/qtest/netdev-socket.c
+++ b/tests/qtest/netdev-socket.c
@@ -18,6 +18,32 @@
 
 #define CONNECTION_TIMEOUT    120
 
+static double connection_timeout(void)
+{
+    double load;
+    int ret = getloadavg(&load, 1);
+
+    /*
+     * If we can't get load data, or load is low because we just started
+     * running, assume load of 1 (we are alone in this system).
+     */
+    if (ret < 1 || load < 1.0) {
+        load = 1.0;
+    }
+    /*
+     * No one wants to wait more than 10 minutes for this test. Higher load?
+     * Too bad.
+     */
+    if (load > 10.0) {
+        fprintf(stderr, "Warning: load %f higher than 10 - test might timeout\n",
+                load);
+        load = 10.0;
+    }
+
+    /* if load is high increase timeout as we might not get a chance to run */
+    return load * CONNECTION_TIMEOUT;
+}
+
 #define EXPECT_STATE(q, e, t)                             \
 do {                                                      \
     char *resp = NULL;                                    \
@@ -31,7 +57,7 @@ do {                                                      \
         if (g_str_equal(resp, e)) {                       \
             break;                                        \
         }                                                 \
-    } while (g_test_timer_elapsed() < CONNECTION_TIMEOUT); \
+    } while (g_test_timer_elapsed() < connection_timeout()); \
     g_assert_cmpstr(resp, ==, e);                         \
     g_free(resp);                                         \
 } while (0)
-- 
MST

Re: [PATCH RFC 0/2] dynamic timeout for tests

[Philippe Mathieu-Daudé]

Hi Michael,

On 7/11/23 10:49, Michael S. Tsirkin wrote:
> so we keep making timeouts longer for CI but one has to stop
> somewhere. netdev socket test recently failed for me again
> even though it's at 2 minutes already.
> here's an experiment for netdev-socket.
> if this works well for a while we can generalize to other
> tests.
> 
> Michael S. Tsirkin (2):
>    osdep: add getloadavg
>    netdev: set timeout depending on loadavg

The 2 problems I see with this approach are:

- Machine load can vary *after* the test is started,
- Test environment isn't really reproducible.

Are those netdev tests only failing due to high load?

Maybe we need to disable them on CI and run them manually...

Regards,

Phil.

Re: [PATCH RFC 0/2] dynamic timeout for tests

[Michael S. Tsirkin]

On Tue, Nov 07, 2023 at 10:57:23AM +0100, Philippe Mathieu-Daudé wrote:
> Hi Michael,
> 
> On 7/11/23 10:49, Michael S. Tsirkin wrote:
> > so we keep making timeouts longer for CI but one has to stop
> > somewhere. netdev socket test recently failed for me again
> > even though it's at 2 minutes already.
> > here's an experiment for netdev-socket.
> > if this works well for a while we can generalize to other
> > tests.
> > 
> > Michael S. Tsirkin (2):
> >    osdep: add getloadavg
> >    netdev: set timeout depending on loadavg
> 
> The 2 problems I see with this approach are:
> 
> - Machine load can vary *after* the test is started,

Then it won't help. But better than increasing timeout
all the time, no?

> - Test environment isn't really reproducible.

But then load intrinsically is.

> Are those netdev tests only failing due to high load?

yep

> Maybe we need to disable them on CI and run them manually...
> 
> Regards,
> 
> Phil.

Most of our tests are like this.

-- 
MST