[Qemu-devel] Help needed: test-qht-par hangs on Travis

[Qemu-devel] Help needed: test-qht-par hangs on Travis

[Eduardo Habkost]

I've noticed QEMU Travis builds are failing recently, and they
seem to happen only on the --enable-gprof jobs.  I have enabled
V=1 and noticed that the jobs are hanging inside test-qht-par.

Example here (look for "/qht/parallel/2threads-0%updates-1s"):

https://travis-ci.org/ehabkost/qemu-hacks/jobs/465081311

Does anybody have any idea why?

-- 
Eduardo

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Emilio G. Cota]

On Fri, Dec 07, 2018 at 18:41:07 -0200, Eduardo Habkost wrote:
> I've noticed QEMU Travis builds are failing recently, and they
> seem to happen only on the --enable-gprof jobs.  I have enabled
> V=1 and noticed that the jobs are hanging inside test-qht-par.
> 
> Example here (look for "/qht/parallel/2threads-0%updates-1s"):
> 
> https://travis-ci.org/ehabkost/qemu-hacks/jobs/465081311
> 
> Does anybody have any idea why?

So if I read that output correctly, it seems that the second
test in qht-par never completes.

Enabling gprof and gcov (as in that build) should just lower
the throughput of the benchmark (test-qht-par invokes qht-bench),
but the duration should be the same (1 second per test, so no need
to wait for 10 minutes).

Can you try re-running the test, after applying the appended patch?
(It disables the "resize" thread.)

Also, does it reliably hang on Travis, or are these hangs
intermittent?

Thanks,

		Emilio
---
diff --git a/tests/test-qht-par.c b/tests/test-qht-par.c
index d8a83caf5c..83ac92e430 100644
--- a/tests/test-qht-par.c
+++ b/tests/test-qht-par.c
@@ -6,7 +6,7 @@
  */
 #include "qemu/osdep.h"

-#define TEST_QHT_STRING "tests/qht-bench 1>/dev/null 2>&1 -R -S0.1 -D10000 -N1 "
+#define TEST_QHT_STRING "tests/qht-bench 1>/dev/null 2>&1 -R "

 static void test_qht(int n_threads, int update_rate, int duration)
 {

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Eduardo Habkost]

On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
> On Fri, Dec 07, 2018 at 18:41:07 -0200, Eduardo Habkost wrote:
> > I've noticed QEMU Travis builds are failing recently, and they
> > seem to happen only on the --enable-gprof jobs.  I have enabled
> > V=1 and noticed that the jobs are hanging inside test-qht-par.
> > 
> > Example here (look for "/qht/parallel/2threads-0%updates-1s"):
> > 
> > https://travis-ci.org/ehabkost/qemu-hacks/jobs/465081311
> > 
> > Does anybody have any idea why?
> 
> So if I read that output correctly, it seems that the second
> test in qht-par never completes.
> 
> Enabling gprof and gcov (as in that build) should just lower
> the throughput of the benchmark (test-qht-par invokes qht-bench),
> but the duration should be the same (1 second per test, so no need
> to wait for 10 minutes).
> 
> Can you try re-running the test, after applying the appended patch?
> (It disables the "resize" thread.)

It is running right now, here:
https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591

> 
> Also, does it reliably hang on Travis, or are these hangs
> intermittent?

It can be reproduced reliably.  qemu.git builds are failing since
Thursday:
https://travis-ci.org/qemu/qemu/builds

> 
> Thanks,
> 
> 		Emilio
> ---
> diff --git a/tests/test-qht-par.c b/tests/test-qht-par.c
> index d8a83caf5c..83ac92e430 100644
> --- a/tests/test-qht-par.c
> +++ b/tests/test-qht-par.c
> @@ -6,7 +6,7 @@
>   */
>  #include "qemu/osdep.h"
> 
> -#define TEST_QHT_STRING "tests/qht-bench 1>/dev/null 2>&1 -R -S0.1 -D10000 -N1 "
> +#define TEST_QHT_STRING "tests/qht-bench 1>/dev/null 2>&1 -R "
> 
>  static void test_qht(int n_threads, int update_rate, int duration)
>  {
> 

-- 
Eduardo

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Emilio G. Cota]

On Mon, Dec 10, 2018 at 14:36:01 -0200, Eduardo Habkost wrote:
> On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
> > Can you try re-running the test, after applying the appended patch?
> > (It disables the "resize" thread.)
> 
> It is running right now, here:
> https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591
> 
> > 
> > Also, does it reliably hang on Travis, or are these hangs
> > intermittent?
> 
> It can be reproduced reliably.  qemu.git builds are failing since
> Thursday:
> https://travis-ci.org/qemu/qemu/builds

I see the build you launched timed out. Can you try the following
patch (after discarding the previous one)? Let's see if just by
disabling the second test we can get the build to move ahead.

Thanks,

		E.
---
diff --git a/tests/test-qht-par.c b/tests/test-qht-par.c
index d8a83caf5c..a916c91ccb 100644
--- a/tests/test-qht-par.c
+++ b/tests/test-qht-par.c
@@ -46,7 +46,6 @@ int main(int argc, char *argv[])

     if (g_test_quick()) {
         g_test_add_func("/qht/parallel/2threads-0%updates-1s", test_2th0u1s);
-        g_test_add_func("/qht/parallel/2threads-20%updates-1s", test_2th20u1s);
     } else {
         g_test_add_func("/qht/parallel/2threads-0%updates-5s", test_2th0u5s);
         g_test_add_func("/qht/parallel/2threads-20%updates-5s", test_2th20u5s);

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Eduardo Habkost]

On Mon, Dec 10, 2018 at 12:07:20PM -0500, Emilio G. Cota wrote:
> On Mon, Dec 10, 2018 at 14:36:01 -0200, Eduardo Habkost wrote:
> > On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
> > > Can you try re-running the test, after applying the appended patch?
> > > (It disables the "resize" thread.)
> > 
> > It is running right now, here:
> > https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591
> > 
> > > 
> > > Also, does it reliably hang on Travis, or are these hangs
> > > intermittent?
> > 
> > It can be reproduced reliably.  qemu.git builds are failing since
> > Thursday:
> > https://travis-ci.org/qemu/qemu/builds
> 
> I see the build you launched timed out. Can you try the following
> patch (after discarding the previous one)? Let's see if just by
> disabling the second test we can get the build to move ahead.

I will try it.  I'm not sure yet if it's the first or the second
test case timing out.  Maybe the "OK\n" we see in the log file is
from another process running in parallel.


> 
> Thanks,
> 
> 		E.
> ---
> diff --git a/tests/test-qht-par.c b/tests/test-qht-par.c
> index d8a83caf5c..a916c91ccb 100644
> --- a/tests/test-qht-par.c
> +++ b/tests/test-qht-par.c
> @@ -46,7 +46,6 @@ int main(int argc, char *argv[])
> 
>      if (g_test_quick()) {
>          g_test_add_func("/qht/parallel/2threads-0%updates-1s", test_2th0u1s);
> -        g_test_add_func("/qht/parallel/2threads-20%updates-1s", test_2th20u1s);
>      } else {
>          g_test_add_func("/qht/parallel/2threads-0%updates-5s", test_2th0u5s);
>          g_test_add_func("/qht/parallel/2threads-20%updates-5s", test_2th20u5s);

-- 
Eduardo

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Eduardo Habkost]

On Mon, Dec 10, 2018 at 03:34:27PM -0200, Eduardo Habkost wrote:
> On Mon, Dec 10, 2018 at 12:07:20PM -0500, Emilio G. Cota wrote:
> > On Mon, Dec 10, 2018 at 14:36:01 -0200, Eduardo Habkost wrote:
> > > On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
> > > > Can you try re-running the test, after applying the appended patch?
> > > > (It disables the "resize" thread.)
> > > 
> > > It is running right now, here:
> > > https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591
> > > 
> > > > 
> > > > Also, does it reliably hang on Travis, or are these hangs
> > > > intermittent?
> > > 
> > > It can be reproduced reliably.  qemu.git builds are failing since
> > > Thursday:
> > > https://travis-ci.org/qemu/qemu/builds
> > 
> > I see the build you launched timed out. Can you try the following
> > patch (after discarding the previous one)? Let's see if just by
> > disabling the second test we can get the build to move ahead.
> 
> I will try it.  I'm not sure yet if it's the first or the second
> test case timing out.  Maybe the "OK\n" we see in the log file is
> from another process running in parallel.

Yeah, I think the first test case is the one hanging:

https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591#L7741

> 
> 
> > 
> > Thanks,
> > 
> > 		E.
> > ---
> > diff --git a/tests/test-qht-par.c b/tests/test-qht-par.c
> > index d8a83caf5c..a916c91ccb 100644
> > --- a/tests/test-qht-par.c
> > +++ b/tests/test-qht-par.c
> > @@ -46,7 +46,6 @@ int main(int argc, char *argv[])
> > 
> >      if (g_test_quick()) {
> >          g_test_add_func("/qht/parallel/2threads-0%updates-1s", test_2th0u1s);
> > -        g_test_add_func("/qht/parallel/2threads-20%updates-1s", test_2th20u1s);
> >      } else {
> >          g_test_add_func("/qht/parallel/2threads-0%updates-5s", test_2th0u5s);
> >          g_test_add_func("/qht/parallel/2threads-20%updates-5s", test_2th20u5s);
> 
> -- 
> Eduardo

-- 
Eduardo

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Eduardo Habkost]

On Mon, Dec 10, 2018 at 05:09:42PM -0200, Eduardo Habkost wrote:
> On Mon, Dec 10, 2018 at 03:34:27PM -0200, Eduardo Habkost wrote:
> > On Mon, Dec 10, 2018 at 12:07:20PM -0500, Emilio G. Cota wrote:
> > > On Mon, Dec 10, 2018 at 14:36:01 -0200, Eduardo Habkost wrote:
> > > > On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
> > > > > Can you try re-running the test, after applying the appended patch?
> > > > > (It disables the "resize" thread.)
> > > > 
> > > > It is running right now, here:
> > > > https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591
> > > > 
> > > > > 
> > > > > Also, does it reliably hang on Travis, or are these hangs
> > > > > intermittent?
> > > > 
> > > > It can be reproduced reliably.  qemu.git builds are failing since
> > > > Thursday:
> > > > https://travis-ci.org/qemu/qemu/builds
> > > 
> > > I see the build you launched timed out. Can you try the following
> > > patch (after discarding the previous one)? Let's see if just by
> > > disabling the second test we can get the build to move ahead.
> > 
> > I will try it.  I'm not sure yet if it's the first or the second
> > test case timing out.  Maybe the "OK\n" we see in the log file is
> > from another process running in parallel.
> 
> Yeah, I think the first test case is the one hanging:
> 
> https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591#L7741

Finally found out what's happening:

https://travis-ci.org/ehabkost/qemu-hacks/builds/478025908

sleep(1) is being interrupted before 1 second has elapsed, and
never exits the loop:

    do {
        remaining = sleep(duration);
    } while (remaining);

-- 
Eduardo

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Paolo Bonzini]

On 10/01/19 21:58, Eduardo Habkost wrote:
> On Mon, Dec 10, 2018 at 05:09:42PM -0200, Eduardo Habkost wrote:
>> On Mon, Dec 10, 2018 at 03:34:27PM -0200, Eduardo Habkost wrote:
>>> On Mon, Dec 10, 2018 at 12:07:20PM -0500, Emilio G. Cota wrote:
>>>> On Mon, Dec 10, 2018 at 14:36:01 -0200, Eduardo Habkost wrote:
>>>>> On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
>>>>>> Can you try re-running the test, after applying the appended patch?
>>>>>> (It disables the "resize" thread.)
>>>>>
>>>>> It is running right now, here:
>>>>> https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591
>>>>>
>>>>>>
>>>>>> Also, does it reliably hang on Travis, or are these hangs
>>>>>> intermittent?
>>>>>
>>>>> It can be reproduced reliably.  qemu.git builds are failing since
>>>>> Thursday:
>>>>> https://travis-ci.org/qemu/qemu/builds
>>>>
>>>> I see the build you launched timed out. Can you try the following
>>>> patch (after discarding the previous one)? Let's see if just by
>>>> disabling the second test we can get the build to move ahead.
>>>
>>> I will try it.  I'm not sure yet if it's the first or the second
>>> test case timing out.  Maybe the "OK\n" we see in the log file is
>>> from another process running in parallel.
>>
>> Yeah, I think the first test case is the one hanging:
>>
>> https://travis-ci.org/ehabkost/qemu-hacks/jobs/466074591#L7741
> 
> Finally found out what's happening:
> 
> https://travis-ci.org/ehabkost/qemu-hacks/builds/478025908
> 
> sleep(1) is being interrupted before 1 second has elapsed, and
> never exits the loop:
> 
>     do {
>         remaining = sleep(duration);
>     } while (remaining);
> 

Great, then the solution is simply to switch to g_usleep!

Paolo

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Eduardo Habkost]

On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
> On Fri, Dec 07, 2018 at 18:41:07 -0200, Eduardo Habkost wrote:
> > I've noticed QEMU Travis builds are failing recently, and they
> > seem to happen only on the --enable-gprof jobs.  I have enabled
> > V=1 and noticed that the jobs are hanging inside test-qht-par.
> > 
> > Example here (look for "/qht/parallel/2threads-0%updates-1s"):
> > 
> > https://travis-ci.org/ehabkost/qemu-hacks/jobs/465081311
> > 
> > Does anybody have any idea why?
> 
> So if I read that output correctly, it seems that the second
> test in qht-par never completes.
> 
> Enabling gprof and gcov (as in that build) should just lower
> the throughput of the benchmark (test-qht-par invokes qht-bench),
[...]

Unrelated question: is there a specific reason why test-qht-par
is written in C using gtest, instead of being just a shell script
that runs qht-bench?

-- 
Eduardo

Re: [Qemu-devel] Help needed: test-qht-par hangs on Travis

[Emilio G. Cota]

On Mon, Dec 10, 2018 at 15:47:15 -0200, Eduardo Habkost wrote:
> On Sun, Dec 09, 2018 at 05:27:38PM -0500, Emilio G. Cota wrote:
> > On Fri, Dec 07, 2018 at 18:41:07 -0200, Eduardo Habkost wrote:
> > > I've noticed QEMU Travis builds are failing recently, and they
> > > seem to happen only on the --enable-gprof jobs.  I have enabled
> > > V=1 and noticed that the jobs are hanging inside test-qht-par.
> > > 
> > > Example here (look for "/qht/parallel/2threads-0%updates-1s"):
> > > 
> > > https://travis-ci.org/ehabkost/qemu-hacks/jobs/465081311
> > > 
> > > Does anybody have any idea why?
> > 
> > So if I read that output correctly, it seems that the second
> > test in qht-par never completes.
> > 
> > Enabling gprof and gcov (as in that build) should just lower
> > the throughput of the benchmark (test-qht-par invokes qht-bench),
> [...]
> 
> Unrelated question: is there a specific reason why test-qht-par
> is written in C using gtest, instead of being just a shell script
> that runs qht-bench?

I didn't know how to integrate a shell script with
gtester, so I went with a C program.

There's a possibility that the use of system(3) here is
what's causing the problem. Can you try running the following
branch on travis?
  https://github.com/cota/qemu/tree/test-qht-par
I moved most of qht-bench into qht-bench.inc.c, so that
both qht-bench.c and test-qht-par.c can use it. This
gets rid of the use of system(3) in test-qht-par.c.

Thanks,

		Emilio