* [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test
@ 2020-11-09 8:07 Philippe Mathieu-Daudé
2020-11-09 9:04 ` Thomas Huth
0 siblings, 1 reply; 6+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-11-09 8:07 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Thomas Huth, Alex Bennée,
Philippe Mathieu-Daudé, Wainer dos Santos Moschetta,
Paolo Bonzini, Pavel Dovgalyuk, Cleber Rosa, Edgar E . Iglesias,
Philippe Mathieu-Daudé
This test is regularly failing on CI :( Do not run it automatically.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/acceptance/boot_linux_console.py | 1 +
tests/acceptance/replay_kernel.py | 1 +
2 files changed, 2 insertions(+)
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 8f433a67f84..010e8790c0f 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -1025,6 +1025,7 @@ def test_m68k_mcf5208evb(self):
tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
+ @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
def test_microblaze_s3adsp1800(self):
"""
:avocado: tags=arch:microblaze
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 00c228382bd..c1f5fa4de71 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -280,6 +280,7 @@ def test_m68k_mcf5208evb(self):
file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
+ @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
def test_microblaze_s3adsp1800(self):
"""
:avocado: tags=arch:microblaze
--
2.26.2
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test
2020-11-09 8:07 [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test Philippe Mathieu-Daudé
@ 2020-11-09 9:04 ` Thomas Huth
2020-11-09 9:09 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 6+ messages in thread
From: Thomas Huth @ 2020-11-09 9:04 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel
Cc: Thomas Huth, Alex Bennée, Richard Henderson,
Wainer dos Santos Moschetta, Paolo Bonzini, Pavel Dovgalyuk,
Cleber Rosa, Edgar E . Iglesias, Philippe Mathieu-Daudé
On 09/11/2020 09.07, Philippe Mathieu-Daudé wrote:
> This test is regularly failing on CI :( Do not run it automatically.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> tests/acceptance/boot_linux_console.py | 1 +
> tests/acceptance/replay_kernel.py | 1 +
> 2 files changed, 2 insertions(+)
>
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 8f433a67f84..010e8790c0f 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -1025,6 +1025,7 @@ def test_m68k_mcf5208evb(self):
> tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
> self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
>
> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> def test_microblaze_s3adsp1800(self):
> """
> :avocado: tags=arch:microblaze
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index 00c228382bd..c1f5fa4de71 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -280,6 +280,7 @@ def test_m68k_mcf5208evb(self):
> file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
>
> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> def test_microblaze_s3adsp1800(self):
> """
> :avocado: tags=arch:microblaze
I think this is a recent regression - it hasn't been failing in the past. We
should first try to find out why it is failing now before sending it to the
@skipUnless nirvana... could you maybe add it to the "Known issues" at
https://wiki.qemu.org/Planning/5.2 instead?
Thomas
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test
2020-11-09 9:04 ` Thomas Huth
@ 2020-11-09 9:09 ` Philippe Mathieu-Daudé
2020-11-09 9:16 ` Thomas Huth
0 siblings, 1 reply; 6+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-11-09 9:09 UTC (permalink / raw)
To: Thomas Huth, qemu-devel
Cc: Thomas Huth, Alex Bennée, Richard Henderson,
Wainer dos Santos Moschetta, Paolo Bonzini, Pavel Dovgalyuk,
Cleber Rosa, Edgar E . Iglesias, Philippe Mathieu-Daudé
On 11/9/20 10:04 AM, Thomas Huth wrote:
> On 09/11/2020 09.07, Philippe Mathieu-Daudé wrote:
>> This test is regularly failing on CI :( Do not run it automatically.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> tests/acceptance/boot_linux_console.py | 1 +
>> tests/acceptance/replay_kernel.py | 1 +
>> 2 files changed, 2 insertions(+)
>>
>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>> index 8f433a67f84..010e8790c0f 100644
>> --- a/tests/acceptance/boot_linux_console.py
>> +++ b/tests/acceptance/boot_linux_console.py
>> @@ -1025,6 +1025,7 @@ def test_m68k_mcf5208evb(self):
>> tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
>> self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
>>
>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>> def test_microblaze_s3adsp1800(self):
>> """
>> :avocado: tags=arch:microblaze
>> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
>> index 00c228382bd..c1f5fa4de71 100644
>> --- a/tests/acceptance/replay_kernel.py
>> +++ b/tests/acceptance/replay_kernel.py
>> @@ -280,6 +280,7 @@ def test_m68k_mcf5208evb(self):
>> file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
>> self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
>>
>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>> def test_microblaze_s3adsp1800(self):
>> """
>> :avocado: tags=arch:microblaze
>
> I think this is a recent regression - it hasn't been failing in the past. We
> should first try to find out why it is failing now before sending it to the
> @skipUnless nirvana... could you maybe add it to the "Known issues" at
> https://wiki.qemu.org/Planning/5.2 instead?
I agree it looks like a regression.
I disagree we should keep broken tests failing the pipeline,
even if we are not using a Gating CI.
I see test_leon3_helenos_uimage has @skip("Test currently broken")
meaning we know there is a problem somewhere, but we don't have
clue/resources to fix the bug, so we demote the test to avoid users
avoid running CI tests.
I'll replace by that.
Regards,
Phil.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test
2020-11-09 9:09 ` Philippe Mathieu-Daudé
@ 2020-11-09 9:16 ` Thomas Huth
2020-11-09 9:23 ` Philippe Mathieu-Daudé
2020-11-09 16:53 ` Wainer dos Santos Moschetta
0 siblings, 2 replies; 6+ messages in thread
From: Thomas Huth @ 2020-11-09 9:16 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel
Cc: Thomas Huth, Alex Bennée, Richard Henderson,
Wainer dos Santos Moschetta, Paolo Bonzini, Pavel Dovgalyuk,
Cleber Rosa, Edgar E . Iglesias, Philippe Mathieu-Daudé
On 09/11/2020 10.09, Philippe Mathieu-Daudé wrote:
> On 11/9/20 10:04 AM, Thomas Huth wrote:
>> On 09/11/2020 09.07, Philippe Mathieu-Daudé wrote:
>>> This test is regularly failing on CI :( Do not run it automatically.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>> tests/acceptance/boot_linux_console.py | 1 +
>>> tests/acceptance/replay_kernel.py | 1 +
>>> 2 files changed, 2 insertions(+)
>>>
>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>> index 8f433a67f84..010e8790c0f 100644
>>> --- a/tests/acceptance/boot_linux_console.py
>>> +++ b/tests/acceptance/boot_linux_console.py
>>> @@ -1025,6 +1025,7 @@ def test_m68k_mcf5208evb(self):
>>> tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
>>> self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
>>>
>>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>>> def test_microblaze_s3adsp1800(self):
>>> """
>>> :avocado: tags=arch:microblaze
>>> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
>>> index 00c228382bd..c1f5fa4de71 100644
>>> --- a/tests/acceptance/replay_kernel.py
>>> +++ b/tests/acceptance/replay_kernel.py
>>> @@ -280,6 +280,7 @@ def test_m68k_mcf5208evb(self):
>>> file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
>>> self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
>>>
>>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>>> def test_microblaze_s3adsp1800(self):
>>> """
>>> :avocado: tags=arch:microblaze
>>
>> I think this is a recent regression - it hasn't been failing in the past. We
>> should first try to find out why it is failing now before sending it to the
>> @skipUnless nirvana... could you maybe add it to the "Known issues" at
>> https://wiki.qemu.org/Planning/5.2 instead?
>
> I agree it looks like a regression.
>
> I disagree we should keep broken tests failing the pipeline,
> even if we are not using a Gating CI.
But what happens if you disable the test at this point in time now? I think
nobody is going to look into this issue anymore since nobody feels
responsible. Thus the bug simply get completely ignored. Please add it at
least the the "Known issues" section.
Thomas
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test
2020-11-09 9:16 ` Thomas Huth
@ 2020-11-09 9:23 ` Philippe Mathieu-Daudé
2020-11-09 16:53 ` Wainer dos Santos Moschetta
1 sibling, 0 replies; 6+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-11-09 9:23 UTC (permalink / raw)
To: Thomas Huth, qemu-devel
Cc: Thomas Huth, Alex Bennée, Richard Henderson,
Wainer dos Santos Moschetta, Paolo Bonzini, Pavel Dovgalyuk,
Cleber Rosa, Edgar E . Iglesias, Philippe Mathieu-Daudé
On 11/9/20 10:16 AM, Thomas Huth wrote:
> On 09/11/2020 10.09, Philippe Mathieu-Daudé wrote:
>> On 11/9/20 10:04 AM, Thomas Huth wrote:
>>> On 09/11/2020 09.07, Philippe Mathieu-Daudé wrote:
>>>> This test is regularly failing on CI :( Do not run it automatically.
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>> tests/acceptance/boot_linux_console.py | 1 +
>>>> tests/acceptance/replay_kernel.py | 1 +
>>>> 2 files changed, 2 insertions(+)
>>>>
>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>>> index 8f433a67f84..010e8790c0f 100644
>>>> --- a/tests/acceptance/boot_linux_console.py
>>>> +++ b/tests/acceptance/boot_linux_console.py
>>>> @@ -1025,6 +1025,7 @@ def test_m68k_mcf5208evb(self):
>>>> tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
>>>> self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
>>>>
>>>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>>>> def test_microblaze_s3adsp1800(self):
>>>> """
>>>> :avocado: tags=arch:microblaze
>>>> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
>>>> index 00c228382bd..c1f5fa4de71 100644
>>>> --- a/tests/acceptance/replay_kernel.py
>>>> +++ b/tests/acceptance/replay_kernel.py
>>>> @@ -280,6 +280,7 @@ def test_m68k_mcf5208evb(self):
>>>> file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
>>>> self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
>>>>
>>>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>>>> def test_microblaze_s3adsp1800(self):
>>>> """
>>>> :avocado: tags=arch:microblaze
>>>
>>> I think this is a recent regression - it hasn't been failing in the past. We
>>> should first try to find out why it is failing now before sending it to the
>>> @skipUnless nirvana... could you maybe add it to the "Known issues" at
>>> https://wiki.qemu.org/Planning/5.2 instead?
>>
>> I agree it looks like a regression.
>>
>> I disagree we should keep broken tests failing the pipeline,
>> even if we are not using a Gating CI.
>
> But what happens if you disable the test at this point in time now? I think
> nobody is going to look into this issue anymore since nobody feels
> responsible. Thus the bug simply get completely ignored.
Corollary: Nobody will use or trust GitLab CI...
> Please add it at
> least the the "Known issues" section.
Will do.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test
2020-11-09 9:16 ` Thomas Huth
2020-11-09 9:23 ` Philippe Mathieu-Daudé
@ 2020-11-09 16:53 ` Wainer dos Santos Moschetta
1 sibling, 0 replies; 6+ messages in thread
From: Wainer dos Santos Moschetta @ 2020-11-09 16:53 UTC (permalink / raw)
To: Thomas Huth, Philippe Mathieu-Daudé, qemu-devel
Cc: Thomas Huth, Alex Bennée, Richard Henderson, Paolo Bonzini,
Pavel Dovgalyuk, Cleber Rosa, Edgar E . Iglesias,
Philippe Mathieu-Daudé
On 11/9/20 7:16 AM, Thomas Huth wrote:
> On 09/11/2020 10.09, Philippe Mathieu-Daudé wrote:
>> On 11/9/20 10:04 AM, Thomas Huth wrote:
>>> On 09/11/2020 09.07, Philippe Mathieu-Daudé wrote:
>>>> This test is regularly failing on CI :( Do not run it automatically.
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>> tests/acceptance/boot_linux_console.py | 1 +
>>>> tests/acceptance/replay_kernel.py | 1 +
>>>> 2 files changed, 2 insertions(+)
>>>>
>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>>>> index 8f433a67f84..010e8790c0f 100644
>>>> --- a/tests/acceptance/boot_linux_console.py
>>>> +++ b/tests/acceptance/boot_linux_console.py
>>>> @@ -1025,6 +1025,7 @@ def test_m68k_mcf5208evb(self):
>>>> tar_hash = 'ac688fd00561a2b6ce1359f9ff6aa2b98c9a570c'
>>>> self.do_test_advcal_2018('07', tar_hash, 'sanity-clause.elf')
>>>>
>>>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>>>> def test_microblaze_s3adsp1800(self):
>>>> """
>>>> :avocado: tags=arch:microblaze
>>>> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
>>>> index 00c228382bd..c1f5fa4de71 100644
>>>> --- a/tests/acceptance/replay_kernel.py
>>>> +++ b/tests/acceptance/replay_kernel.py
>>>> @@ -280,6 +280,7 @@ def test_m68k_mcf5208evb(self):
>>>> file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
>>>> self.do_test_advcal_2018(file_path, 'sanity-clause.elf')
>>>>
>>>> + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>>>> def test_microblaze_s3adsp1800(self):
>>>> """
>>>> :avocado: tags=arch:microblaze
>>> I think this is a recent regression - it hasn't been failing in the past. We
>>> should first try to find out why it is failing now before sending it to the
>>> @skipUnless nirvana... could you maybe add it to the "Known issues" at
>>> https://wiki.qemu.org/Planning/5.2 instead?
>> I agree it looks like a regression.
>>
>> I disagree we should keep broken tests failing the pipeline,
>> even if we are not using a Gating CI.
> But what happens if you disable the test at this point in time now? I think
> nobody is going to look into this issue anymore since nobody feels
> responsible. Thus the bug simply get completely ignored. Please add it at
> least the the "Known issues" section.
Would make sense do the following?
1. Introduce a new tag (e.g. "regression") to mark tests which are
currently broken due regression bugs
2. Filter those tests out of the regular acceptance jobs
3. Add a new job to run only those tests
3.1. Use the allow_failure [1] property so that this job won't change
the overall testing status
Then if a bug is fixed and the test start to pass again, remove the
"regression" tag. Otherwise if the bug is still present on occasion of
QEMU release then it is definitively marked to skip.
[1] https://docs.gitlab.com/ee/ci/yaml/#allow_failure
- Wainer
>
> Thomas
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-11-09 16:54 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-11-09 8:07 [RFC PATCH-for-5.2] tests/acceptance: Disable Spartan-3A DSP 1800A test Philippe Mathieu-Daudé
2020-11-09 9:04 ` Thomas Huth
2020-11-09 9:09 ` Philippe Mathieu-Daudé
2020-11-09 9:16 ` Thomas Huth
2020-11-09 9:23 ` Philippe Mathieu-Daudé
2020-11-09 16:53 ` Wainer dos Santos Moschetta
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).