Re: [Qemu-devel] [RFC PATCH 2/2] tests/Makefile: comment out flakey tests

[Thomas Huth <thuth@redhat.com>]

On 18.05.2018 20:31, Peter Maydell wrote:
> On 18 May 2018 at 10:14, Alex Bennée <alex.bennee@linaro.org> wrote:
>> The following tests keep showing up in failed Travis runs:
>>
>>   - test-aio
>>   - rcutorture
>>   - tpm-crb-test
>>   - tpm-tis-test
>>
>> I suspect it is load that causes the problems but they really need to
>> be fixed properly.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> Another flaky test for the collection:
> 
> TEST: tests/boot-serial-test... (pid=25144)
>   /sparc64/boot-serial/sun4u:                                          **
> ERROR:/home/petmay01/linaro/qemu-for-merges/tests/boot-serial-test.c:140:check_guest_output:
> assertion failed: (output_ok)
> FAIL
> 
> Probably another "overly optimistic timeout" setting. (Failed
> for me on x86-64 host just now.)

That test normally finishes within 3 seconds on my machine. The test
timeout is 60 seconds. How much load did you have on that machine to go
from 3s to 60s ?

And even if we increase the timeout, how to find a good value here? I
think we rather need a "no-timeout" switch where we can tell the tests
to not use timeouts and rather run forever instead, until they really
finished? So in normal interactive mode, we'd run with timeouts, but
when running on a loaded builder machine, you'd enable that "no-timeout"
switch to make sure to not run in such "early" timeouts.

 Thomas