[PATCH] tests/avocado/replay_kernel: Mark the test_x86_64_pc as flaky

[PATCH] tests/avocado/replay_kernel: Mark the test_x86_64_pc as flaky

[Thomas Huth]

It's failing very often, so don't run this by default anymore
until it gets fixed.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/avocado/replay_kernel.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
index a18610542e..53cb7e5091 100644
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -81,7 +81,8 @@ def run_rr(self, kernel_path, kernel_command_line, console_pattern,
         logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
 
 class ReplayKernelNormal(ReplayKernelBase):
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
-- 
2.41.0

Re: [PATCH] tests/avocado/replay_kernel: Mark the test_x86_64_pc as flaky

[Peter Maydell]

On Tue, 14 Nov 2023 at 15:30, Thomas Huth <thuth@redhat.com> wrote:
>
> It's failing very often, so don't run this by default anymore
> until it gets fixed.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/avocado/replay_kernel.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
> index a18610542e..53cb7e5091 100644
> --- a/tests/avocado/replay_kernel.py
> +++ b/tests/avocado/replay_kernel.py
> @@ -81,7 +81,8 @@ def run_rr(self, kernel_path, kernel_command_line, console_pattern,
>          logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
>
>  class ReplayKernelNormal(ReplayKernelBase):
> -    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
> +
> +    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
>      def test_x86_64_pc(self):
>          """
>          :avocado: tags=arch:x86_64

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

We should replace all of the "skip if GITLAB_CI" checks
with "skip on QEMU_TEST_FLAKY_TESTS" (or other check as
appropriate), I think.

thanks
-- PMM

Re: [PATCH] tests/avocado/replay_kernel: Mark the test_x86_64_pc as flaky

[Philippe Mathieu-Daudé]

On 14/11/23 16:30, Thomas Huth wrote:
> It's failing very often, so don't run this by default anymore
> until it gets fixed.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   tests/avocado/replay_kernel.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
> index a18610542e..53cb7e5091 100644
> --- a/tests/avocado/replay_kernel.py
> +++ b/tests/avocado/replay_kernel.py
> @@ -81,7 +81,8 @@ def run_rr(self, kernel_path, kernel_command_line, console_pattern,
>           logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
>   
>   class ReplayKernelNormal(ReplayKernelBase):
> -    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
> +
> +    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
>       def test_x86_64_pc(self):
>           """
>           :avocado: tags=arch:x86_64

Hmm I suspect https://gitlab.com/qemu-project/qemu/-/issues/1961 which
has a fix ready:
https://lore.kernel.org/qemu-devel/20231110170831.185001-1-richard.henderson@linaro.org/

Re: [PATCH] tests/avocado/replay_kernel: Mark the test_x86_64_pc as flaky

[Thomas Huth]

On 14/11/2023 17.36, Philippe Mathieu-Daudé wrote:
> On 14/11/23 16:30, Thomas Huth wrote:
>> It's failing very often, so don't run this by default anymore
>> until it gets fixed.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   tests/avocado/replay_kernel.py | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
>> index a18610542e..53cb7e5091 100644
>> --- a/tests/avocado/replay_kernel.py
>> +++ b/tests/avocado/replay_kernel.py
>> @@ -81,7 +81,8 @@ def run_rr(self, kernel_path, kernel_command_line, 
>> console_pattern,
>>           logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
>>   class ReplayKernelNormal(ReplayKernelBase):
>> -    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
>> +
>> +    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets 
>> stuck')
>>       def test_x86_64_pc(self):
>>           """
>>           :avocado: tags=arch:x86_64
> 
> Hmm I suspect https://gitlab.com/qemu-project/qemu/-/issues/1961 which
> has a fix ready:
> https://lore.kernel.org/qemu-devel/20231110170831.185001-1-richard.henderson@linaro.org/

I just gave it a try, too, and it does not fix the issue for me.

Is this test still working for anybody?

  Thomas

Re: [PATCH] tests/avocado/replay_kernel: Mark the test_x86_64_pc as flaky

[Thomas Huth]

On 14/11/2023 17.22, Peter Maydell wrote:
> On Tue, 14 Nov 2023 at 15:30, Thomas Huth <thuth@redhat.com> wrote:
>>
>> It's failing very often, so don't run this by default anymore
>> until it gets fixed.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   tests/avocado/replay_kernel.py | 3 ++-
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
>> index a18610542e..53cb7e5091 100644
>> --- a/tests/avocado/replay_kernel.py
>> +++ b/tests/avocado/replay_kernel.py
>> @@ -81,7 +81,8 @@ def run_rr(self, kernel_path, kernel_command_line, console_pattern,
>>           logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
>>
>>   class ReplayKernelNormal(ReplayKernelBase):
>> -    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
>> +
>> +    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
>>       def test_x86_64_pc(self):
>>           """
>>           :avocado: tags=arch:x86_64
> 
> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
> 
> We should replace all of the "skip if GITLAB_CI" checks
> with "skip on QEMU_TEST_FLAKY_TESTS" (or other check as
> appropriate), I think.

Yes ... the question is whether they just time out because gitlab is too 
slow (in that case we should increase the timeout), or whether they are 
really flaky ... needs some time for investigation, I guess.

  Thomas