* [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 13:43 [PATCH v4 0/6] testing/next: avocado, docs, gitlab Alex Bennée
@ 2023-04-17 13:43 ` Alex Bennée
2023-04-17 16:50 ` John Snow
` (3 more replies)
2023-04-17 13:43 ` [PATCH v4 2/6] tests/avocado: use the new snapshots for testing Alex Bennée
` (5 subsequent siblings)
6 siblings, 4 replies; 23+ messages in thread
From: Alex Bennée @ 2023-04-17 13:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Alex Bennée, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
John Snow, Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Kautuk Consul, Hariharan T S
From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.
Bump up the avocado-framework version to 101.0.
Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
---
v2
- limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
---
tests/Makefile.include | 18 +++++++++++-------
tests/requirements.txt | 2 +-
2 files changed, 12 insertions(+), 8 deletions(-)
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 9422ddaece..a4de0ad5a2 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
# download all vm images, according to defined targets
get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
+JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
+
check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
- $(call quiet-command, \
- $(TESTS_PYTHON) -m avocado \
- --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
- $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
- --filter-by-tags-include-empty-key) \
- $(AVOCADO_CMDLINE_TAGS) \
- $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
+ $(call quiet-command, \
+ $(TESTS_PYTHON) -m avocado \
+ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+ $(if $(AVOCADO_TAGS),, \
+ --filter-by-tags-include-empty \
+ --filter-by-tags-include-empty-key) \
+ --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
+ $(AVOCADO_CMDLINE_TAGS) \
+ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
"AVOCADO", "tests/avocado")
check-acceptance-deprecated-warning:
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 0ba561b6bd..a6f73da681 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -2,5 +2,5 @@
# in the tests/venv Python virtual environment. For more info,
# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
# Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
+avocado-framework==101.0
pycdlib==1.11.0
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 13:43 ` [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0 Alex Bennée
@ 2023-04-17 16:50 ` John Snow
2023-04-17 17:44 ` Alex Bennée
2023-04-21 3:56 ` Cleber Rosa
2023-04-18 7:10 ` Thomas Huth
` (2 subsequent siblings)
3 siblings, 2 replies; 23+ messages in thread
From: John Snow @ 2023-04-17 16:50 UTC (permalink / raw)
To: Alex Bennée, Cleber Rosa, Jan Richter,
Philippe Mathieu-Daudé
Cc: qemu-devel, Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
> tests/Makefile.include | 18 +++++++++++-------
> tests/requirements.txt | 2 +-
> 2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> # download all vm images, according to defined targets
> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> - $(call quiet-command, \
> - $(TESTS_PYTHON) -m avocado \
> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> - --filter-by-tags-include-empty-key) \
> - $(AVOCADO_CMDLINE_TAGS) \
> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> + $(call quiet-command, \
> + $(TESTS_PYTHON) -m avocado \
> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> + $(if $(AVOCADO_TAGS),, \
> + --filter-by-tags-include-empty \
> + --filter-by-tags-include-empty-key) \
> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
> + $(AVOCADO_CMDLINE_TAGS) \
> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> "AVOCADO", "tests/avocado")
>
> check-acceptance-deprecated-warning:
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 0ba561b6bd..a6f73da681 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,5 +2,5 @@
> # in the tests/venv Python virtual environment. For more info,
> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> # Note that qemu.git/python/ is always implicitly installed.
> -avocado-framework==88.1
> +avocado-framework==101.0
> pycdlib==1.11.0
> --
> 2.39.2
>
I thought there were test failures that prohibited us from bumping the
Avocado-Framework version. Did those get rectified recently?
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 16:50 ` John Snow
@ 2023-04-17 17:44 ` Alex Bennée
2023-04-20 9:18 ` Philippe Mathieu-Daudé
2023-04-21 3:56 ` Cleber Rosa
1 sibling, 1 reply; 23+ messages in thread
From: Alex Bennée @ 2023-04-17 17:44 UTC (permalink / raw)
To: John Snow
Cc: Cleber Rosa, Jan Richter, Philippe Mathieu-Daudé, qemu-devel,
Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
John Snow <jsnow@redhat.com> writes:
> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>> tests/Makefile.include | 18 +++++++++++-------
>> tests/requirements.txt | 2 +-
>> 2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>> # download all vm images, according to defined targets
>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>
>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>> +
>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> - $(call quiet-command, \
>> - $(TESTS_PYTHON) -m avocado \
>> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> - --filter-by-tags-include-empty-key) \
>> - $(AVOCADO_CMDLINE_TAGS) \
>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> + $(call quiet-command, \
>> + $(TESTS_PYTHON) -m avocado \
>> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> + $(if $(AVOCADO_TAGS),, \
>> + --filter-by-tags-include-empty \
>> + --filter-by-tags-include-empty-key) \
>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
>> + $(AVOCADO_CMDLINE_TAGS) \
>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> "AVOCADO", "tests/avocado")
>>
>> check-acceptance-deprecated-warning:
>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>> index 0ba561b6bd..a6f73da681 100644
>> --- a/tests/requirements.txt
>> +++ b/tests/requirements.txt
>> @@ -2,5 +2,5 @@
>> # in the tests/venv Python virtual environment. For more info,
>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>> # Note that qemu.git/python/ is always implicitly installed.
>> -avocado-framework==88.1
>> +avocado-framework==101.0
>> pycdlib==1.11.0
>> --
>> 2.39.2
>>
>
> I thought there were test failures that prohibited us from bumping the
> Avocado-Framework version. Did those get rectified recently?
I'm not seeing any with your patch applied.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 17:44 ` Alex Bennée
@ 2023-04-20 9:18 ` Philippe Mathieu-Daudé
2023-04-20 16:54 ` John Snow
2023-04-21 4:04 ` Cleber Rosa
0 siblings, 2 replies; 23+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-04-20 9:18 UTC (permalink / raw)
To: Alex Bennée, John Snow
Cc: Cleber Rosa, Jan Richter, qemu-devel, Peter Maydell,
Bastian Koppelmann, Markus Armbruster, Beraldo Leal, Thomas Huth,
Stefan Hajnoczi, Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
On 17/4/23 19:44, Alex Bennée wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
>> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the
>>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due
>>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado
>>> code would start downloading a fresh image from the internet
>>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>> ---
>>> tests/Makefile.include | 18 +++++++++++-------
>>> tests/requirements.txt | 2 +-
>>> 2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>> # download all vm images, according to defined targets
>>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>>
>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>>> +
>>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> - $(call quiet-command, \
>>> - $(TESTS_PYTHON) -m avocado \
>>> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>> - --filter-by-tags-include-empty-key) \
>>> - $(AVOCADO_CMDLINE_TAGS) \
>>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> + $(call quiet-command, \
>>> + $(TESTS_PYTHON) -m avocado \
>>> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> + $(if $(AVOCADO_TAGS),, \
>>> + --filter-by-tags-include-empty \
>>> + --filter-by-tags-include-empty-key) \
>>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
>>> + $(AVOCADO_CMDLINE_TAGS) \
>>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> "AVOCADO", "tests/avocado")
>>>
>>> check-acceptance-deprecated-warning:
>>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>> index 0ba561b6bd..a6f73da681 100644
>>> --- a/tests/requirements.txt
>>> +++ b/tests/requirements.txt
>>> @@ -2,5 +2,5 @@
>>> # in the tests/venv Python virtual environment. For more info,
>>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>> # Note that qemu.git/python/ is always implicitly installed.
>>> -avocado-framework==88.1
>>> +avocado-framework==101.0
>>> pycdlib==1.11.0
>>> --
>>> 2.39.2
>>>
>>
>> I thought there were test failures that prohibited us from bumping the
>> Avocado-Framework version. Did those get rectified recently?
No, still not working on Darwin. At this point I'm thinking at sending
a patch clarifying testing QEMU with Avocado is not supported on Darwin.
> I'm not seeing any with your patch applied.
Are you using running the Cirrus-CI jobs?
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-20 9:18 ` Philippe Mathieu-Daudé
@ 2023-04-20 16:54 ` John Snow
2023-04-21 4:09 ` Cleber Rosa
2023-04-21 4:04 ` Cleber Rosa
1 sibling, 1 reply; 23+ messages in thread
From: John Snow @ 2023-04-20 16:54 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Alex Bennée, Cleber Rosa, Jan Richter, qemu-devel,
Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
[-- Attachment #1: Type: text/plain, Size: 4876 bytes --]
On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé <philmd@linaro.org>
wrote:
> On 17/4/23 19:44, Alex Bennée wrote:
> >
> > John Snow <jsnow@redhat.com> writes:
> >
> >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >>>
> >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>>
> >>> Avocado version 101.0 has a fix to re-compute the checksum
> >>> of an asset file if the algorithm used in the *-CHECKSUM
> >>> file isn't the same as the one being passed to it by the
> >>> avocado user (i.e. the avocado_qemu python module).
> >>> In the earlier avocado versions this fix wasn't there due
> >>> to which if the checksum wouldn't match the earlier
> >>> checksum (calculated by a different algorithm), the avocado
> >>> code would start downloading a fresh image from the internet
> >>> URL thus making the test-cases take longer to execute.
> >>>
> >>> Bump up the avocado-framework version to 101.0.
> >>>
> >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> >>>
> >>> ---
> >>> v2
> >>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> >>> ---
> >>> tests/Makefile.include | 18 +++++++++++-------
> >>> tests/requirements.txt | 2 +-
> >>> 2 files changed, 12 insertions(+), 8 deletions(-)
> >>>
> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>> index 9422ddaece..a4de0ad5a2 100644
> >>> --- a/tests/Makefile.include
> >>> +++ b/tests/Makefile.include
> >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> >>> # download all vm images, according to defined targets
> >>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%,
> $(FEDORA_31_DOWNLOAD))
> >>>
> >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
> -j%,$(MAKEFLAGS))))
> >>> +
> >>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> >>> - $(call quiet-command, \
> >>> - $(TESTS_PYTHON) -m avocado \
> >>> - --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> >>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> >>> - --filter-by-tags-include-empty-key) \
> >>> - $(AVOCADO_CMDLINE_TAGS) \
> >>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> >>> + $(call quiet-command,
> \
> >>> + $(TESTS_PYTHON) -m avocado
> \
> >>> + --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> >>> + $(if $(AVOCADO_TAGS),,
> \
> >>> + --filter-by-tags-include-empty
> \
> >>> + --filter-by-tags-include-empty-key)
> \
> >>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> \
> >>> + $(AVOCADO_CMDLINE_TAGS)
> \
> >>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),
> \
> >>> "AVOCADO", "tests/avocado")
> >>>
> >>> check-acceptance-deprecated-warning:
> >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
> >>> index 0ba561b6bd..a6f73da681 100644
> >>> --- a/tests/requirements.txt
> >>> +++ b/tests/requirements.txt
> >>> @@ -2,5 +2,5 @@
> >>> # in the tests/venv Python virtual environment. For more info,
> >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >>> # Note that qemu.git/python/ is always implicitly installed.
> >>> -avocado-framework==88.1
> >>> +avocado-framework==101.0
> >>> pycdlib==1.11.0
> >>> --
> >>> 2.39.2
> >>>
> >>
> >> I thought there were test failures that prohibited us from bumping the
> >> Avocado-Framework version. Did those get rectified recently?
>
> No, still not working on Darwin. At this point I'm thinking at sending
> a patch clarifying testing QEMU with Avocado is not supported on Darwin.
>
What's broken on Darwin?
Is it broken with avocado-framework==88.1 too? (Is this a regression?)
We might need to consider entirely separate issues:
(1) We need to upgrade avocado-framework to a supported LTS version >= v90
for dependency harmony with qemu.git/python testing
(2) We need to upgrade avocado-framework to >= v101.0 to fix the stated
checksum issue in this patch
(3) We would like avocado tests to work on Darwin. (Have they ever worked?
When did they break? etc.)
My understanding is that this patch helps with 1 and 2 but not 3, but if
Darwin support isn't a regression, this is still a step forward.
> > I'm not seeing any with your patch applied.
>
> Are you using running the Cirrus-CI jobs?
>
>
[-- Attachment #2: Type: text/html, Size: 7748 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-20 16:54 ` John Snow
@ 2023-04-21 4:09 ` Cleber Rosa
2023-04-21 10:07 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 23+ messages in thread
From: Cleber Rosa @ 2023-04-21 4:09 UTC (permalink / raw)
To: John Snow, Philippe Mathieu-Daudé
Cc: Alex Bennée, Jan Richter, qemu-devel, Peter Maydell,
Bastian Koppelmann, Markus Armbruster, Beraldo Leal, Thomas Huth,
Stefan Hajnoczi, Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
On 4/20/23 12:54, John Snow wrote:
>
>
> On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé
> <philmd@linaro.org> wrote:
>
> On 17/4/23 19:44, Alex Bennée wrote:
> >
> > John Snow <jsnow@redhat.com> writes:
> >
> >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée
> <alex.bennee@linaro.org> wrote:
> >>>
> >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>>
> >>> Avocado version 101.0 has a fix to re-compute the checksum
> >>> of an asset file if the algorithm used in the *-CHECKSUM
> >>> file isn't the same as the one being passed to it by the
> >>> avocado user (i.e. the avocado_qemu python module).
> >>> In the earlier avocado versions this fix wasn't there due
> >>> to which if the checksum wouldn't match the earlier
> >>> checksum (calculated by a different algorithm), the avocado
> >>> code would start downloading a fresh image from the internet
> >>> URL thus making the test-cases take longer to execute.
> >>>
> >>> Bump up the avocado-framework version to 101.0.
> >>>
> >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> >>>
> >>> ---
> >>> v2
> >>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> >>> ---
> >>> tests/Makefile.include | 18 +++++++++++-------
> >>> tests/requirements.txt | 2 +-
> >>> 2 files changed, 12 insertions(+), 8 deletions(-)
> >>>
> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>> index 9422ddaece..a4de0ad5a2 100644
> >>> --- a/tests/Makefile.include
> >>> +++ b/tests/Makefile.include
> >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> >>> # download all vm images, according to defined targets
> >>> get-vm-images: check-venv $(patsubst
> %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
> >>>
> >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
> -j%,$(MAKEFLAGS))))
> >>> +
> >>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> >>> - $(call quiet-command, \
> >>> - $(TESTS_PYTHON) -m avocado \
> >>> - --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> >>> - $(if $(AVOCADO_TAGS),,
> --filter-by-tags-include-empty \
> >>> - --filter-by-tags-include-empty-key) \
> >>> - $(AVOCADO_CMDLINE_TAGS) \
> >>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> >>> + $(call quiet-command, \
> >>> + $(TESTS_PYTHON) -m avocado
> \
> >>> + --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> >>> + $(if $(AVOCADO_TAGS),,
> \
> >>> + --filter-by-tags-include-empty \
> >>> + --filter-by-tags-include-empty-key) \
> >>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> \
> >>> + $(AVOCADO_CMDLINE_TAGS)
> \
> >>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),
> \
> >>> "AVOCADO", "tests/avocado")
> >>>
> >>> check-acceptance-deprecated-warning:
> >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
> >>> index 0ba561b6bd..a6f73da681 100644
> >>> --- a/tests/requirements.txt
> >>> +++ b/tests/requirements.txt
> >>> @@ -2,5 +2,5 @@
> >>> # in the tests/venv Python virtual environment. For more info,
> >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >>> # Note that qemu.git/python/ is always implicitly installed.
> >>> -avocado-framework==88.1
> >>> +avocado-framework==101.0
> >>> pycdlib==1.11.0
> >>> --
> >>> 2.39.2
> >>>
> >>
> >> I thought there were test failures that prohibited us from
> bumping the
> >> Avocado-Framework version. Did those get rectified recently?
>
> No, still not working on Darwin. At this point I'm thinking at sending
> a patch clarifying testing QEMU with Avocado is not supported on
> Darwin.
>
>
> What's broken on Darwin?
>
Hi John,
IIRC, the main issue is that the legacy runner (the one active in 88.1)
pickles some stuff that do not play nice under Darwin. It has never (to
the best of my knowledge) worked under Darwin.
> Is it broken with avocado-framework==88.1 too? (Is this a regression?)
>
It was already broken with avocado-framework==88.1, but it should mostly
work with 101.0 (see previous reply).
> We might need to consider entirely separate issues:
>
> (1) We need to upgrade avocado-framework to a supported LTS version >=
> v90 for dependency harmony with qemu.git/python testing
>
> (2) We need to upgrade avocado-framework to >= v101.0 to fix the
> stated checksum issue in this patch
>
> (3) We would like avocado tests to work on Darwin. (Have they ever
> worked? When did they break? etc.)
>
> My understanding is that this patch helps with 1 and 2 but not 3, but
> if Darwin support isn't a regression, this is still a step forward.
>
To the best of my knowledge, this patch would help with 1, 2 and even 3.
Cheers,
- Cleber.
>
> > I'm not seeing any with your patch applied.
>
> Are you using running the Cirrus-CI jobs?
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-21 4:09 ` Cleber Rosa
@ 2023-04-21 10:07 ` Philippe Mathieu-Daudé
2023-04-21 10:27 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 23+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-04-21 10:07 UTC (permalink / raw)
To: Cleber Rosa, John Snow
Cc: Alex Bennée, Jan Richter, qemu-devel, Peter Maydell,
Bastian Koppelmann, Markus Armbruster, Beraldo Leal, Thomas Huth,
Stefan Hajnoczi, Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
On 21/4/23 06:09, Cleber Rosa wrote:
>
> On 4/20/23 12:54, John Snow wrote:
>>
>>
>> On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé
>> <philmd@linaro.org> wrote:
>>
>> On 17/4/23 19:44, Alex Bennée wrote:
>> >
>> > John Snow <jsnow@redhat.com> writes:
>> >
>> >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée
>> <alex.bennee@linaro.org> wrote:
>> >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>> >>> index 0ba561b6bd..a6f73da681 100644
>> >>> --- a/tests/requirements.txt
>> >>> +++ b/tests/requirements.txt
>> >>> @@ -2,5 +2,5 @@
>> >>> # in the tests/venv Python virtual environment. For more info,
>> >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>> >>> # Note that qemu.git/python/ is always implicitly installed.
>> >>> -avocado-framework==88.1
>> >>> +avocado-framework==101.0
>> >>> pycdlib==1.11.0
>> >>> --
>> >>> 2.39.2
>> >>>
>> >>
>> >> I thought there were test failures that prohibited us from
>> bumping the
>> >> Avocado-Framework version. Did those get rectified recently?
>>
>> No, still not working on Darwin. At this point I'm thinking at
>> sending
>> a patch clarifying testing QEMU with Avocado is not supported on
>> Darwin.
>>
>>
>> What's broken on Darwin?
>>
> Hi John,
>
> IIRC, the main issue is that the legacy runner (the one active in 88.1)
> pickles some stuff that do not play nice under Darwin. It has never (to
> the best of my knowledge) worked under Darwin.
Correct.
>> Is it broken with avocado-framework==88.1 too? (Is this a regression?)
>>
> It was already broken with avocado-framework==88.1,
Correct.
> but it should mostly
> work with 101.0 (see previous reply).
>
>> We might need to consider entirely separate issues:
>>
>> (1) We need to upgrade avocado-framework to a supported LTS version >=
>> v90 for dependency harmony with qemu.git/python testing
>>
>> (2) We need to upgrade avocado-framework to >= v101.0 to fix the
>> stated checksum issue in this patch
>>
>> (3) We would like avocado tests to work on Darwin. (Have they ever
>> worked? When did they break? etc.)
>>
>> My understanding is that this patch helps with 1 and 2 but not 3, but
>> if Darwin support isn't a regression, this is still a step forward.
>>
> To the best of my knowledge, this patch would help with 1, 2 and even 3.
As of today using v8.0.0:
$ make check-avocado
GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc
MKDIR /Users/philmd/source/qemu/build/tests/results
AVOCADO Downloading avocado tests VM image for aarch64
The image was downloaded:
Provider Version Architecture File
fedora 31 aarch64
/Users/philmd/avocado/data/cache/by_location/4f156e531446a679cbfe13caef8b7c9f9f79aafa/Fedora-Cloud-Base-31-1.9.aarch64.qcow2
AVOCADO tests/avocado
Fetching asset from
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf
Fetching asset from
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0
Fetching asset from
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0
Fetching asset from
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0
Fetching asset from
tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_alpine_virt_tcg_gic_max
Fetching asset from
tests/avocado/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt
Fetching asset from
tests/avocado/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt
JOB ID : 142a0b2d3960c7daccd657781d52f814e1617392
JOB LOG :
/Users/philmd/source/qemu/build/tests/results/job-2023-04-21T11.35-142a0b2/job.log
Avocado crashed: TypeError: cannot pickle '_thread.RLock' object
Traceback (most recent call last):
File
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/job.py",
line 632, in run_tests
summary |= suite.run(self)
File
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/suite.py",
line 311, in run
return self.runner.run_suite(job, self)
File
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py",
line 390, in run_suite
if not self.run_test(job, test_factory, queue, summary,
File
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py",
line 170, in run_test
proc.start()
File
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py",
line 121, in start
self._popen = self._Popen(self)
File
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py",
line 288, in _Popen
return Popen(process_obj)
File
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py",
line 32, in __init__
super().__init__(process_obj)
File
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py",
line 19, in __init__
self._launch(process_obj)
File
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py",
line 47, in _launch
reduction.dump(process_obj, fp)
File
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py",
line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object
Please include the traceback info and command line used on your bug report
Report bugs visiting https://github.com/avocado-framework/avocado/issues/new
make: *** [check-avocado] Error 4
This is https://github.com/avocado-framework/avocado/issues/4888
which is still opened.
Manually using "avocado run --test-runner=nrunner" helps running the
tests but the console output is never displayed (see [*]) and once
finished Avocado lets various QEMU processes running in detached
mode, wasting resources.
[*]
https://lore.kernel.org/qemu-devel/1d0baf8b-c757-265c-b206-07ca3f218b2a@linaro.org/
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-21 10:07 ` Philippe Mathieu-Daudé
@ 2023-04-21 10:27 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 23+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-04-21 10:27 UTC (permalink / raw)
To: Cleber Rosa, John Snow
Cc: Alex Bennée, Jan Richter, qemu-devel, Peter Maydell,
Bastian Koppelmann, Markus Armbruster, Beraldo Leal, Thomas Huth,
Stefan Hajnoczi, Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
On 21/4/23 12:07, Philippe Mathieu-Daudé wrote:
> On 21/4/23 06:09, Cleber Rosa wrote:
> Manually using "avocado run --test-runner=nrunner" helps running the
> tests but the console output is never displayed (see [*]) and once
> finished Avocado lets various QEMU processes running in detached
> mode, wasting resources.=
Using this patch:
$ ./tests/venv/bin/avocado --show=console -V run -t arch:aarch64
tests/avocado/boot_*
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
ERROR
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_xlnx_versal_virt:
ERROR
Not asking for the console output at least show the test being
processed (similar to old runner):
$ ./tests/venv/bin/avocado run -t machine:raspi3b tests/avocado/boot_*
Fetching asset from
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf
JOB ID : 1c963c12f7f07e109ef8480ddf204f02e61f8b8d
JOB LOG :
/Users/philmd/avocado/job-results/job-2023-04-21T12.24-1c963c1/job.log
(1/1)
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
STARTED
(1/1)
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
ERROR: ConnectError: Failed to establish session: EOFError\n Exit code:
1\n Command: ./qemu-system-aarch64 -display none -vga none -chardev
socket,id=mon,fd=11 -mon chardev=mon,mode=control -machine raspi3b
-serial null -chardev socket,id=console,path=/var/folders... (0.31 s)
RESULTS : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
CANCEL 0
JOB TIME : 2.20 s
Test summary:
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
ERROR
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-20 9:18 ` Philippe Mathieu-Daudé
2023-04-20 16:54 ` John Snow
@ 2023-04-21 4:04 ` Cleber Rosa
1 sibling, 0 replies; 23+ messages in thread
From: Cleber Rosa @ 2023-04-21 4:04 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, Alex Bennée, John Snow
Cc: Jan Richter, qemu-devel, Peter Maydell, Bastian Koppelmann,
Markus Armbruster, Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
On 4/20/23 05:18, Philippe Mathieu-Daudé wrote:
> On 17/4/23 19:44, Alex Bennée wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>>> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org>
>>> wrote:
>>>>
>>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>>
>>>> Avocado version 101.0 has a fix to re-compute the checksum
>>>> of an asset file if the algorithm used in the *-CHECKSUM
>>>> file isn't the same as the one being passed to it by the
>>>> avocado user (i.e. the avocado_qemu python module).
>>>> In the earlier avocado versions this fix wasn't there due
>>>> to which if the checksum wouldn't match the earlier
>>>> checksum (calculated by a different algorithm), the avocado
>>>> code would start downloading a fresh image from the internet
>>>> URL thus making the test-cases take longer to execute.
>>>>
>>>> Bump up the avocado-framework version to 101.0.
>>>>
>>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>>
>>>> ---
>>>> v2
>>>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>>> ---
>>>> tests/Makefile.include | 18 +++++++++++-------
>>>> tests/requirements.txt | 2 +-
>>>> 2 files changed, 12 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index 9422ddaece..a4de0ad5a2 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>> # download all vm images, according to defined targets
>>>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%,
>>>> $(FEDORA_31_DOWNLOAD))
>>>>
>>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
>>>> -j%,$(MAKEFLAGS))))
>>>> +
>>>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>>> - $(call quiet-command, \
>>>> - $(TESTS_PYTHON) -m avocado \
>>>> - --show=$(AVOCADO_SHOW) run
>>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>>> - --filter-by-tags-include-empty-key) \
>>>> - $(AVOCADO_CMDLINE_TAGS) \
>>>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>>> + $(call quiet-command, \
>>>> + $(TESTS_PYTHON) -m
>>>> avocado \
>>>> + --show=$(AVOCADO_SHOW) run
>>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>>> + $(if $(AVOCADO_TAGS),, \
>>>> + --filter-by-tags-include-empty \
>>>> + --filter-by-tags-include-empty-key) \
>>>> + --max-parallel-tasks
>>>> $(JOBS_OPTION:-j%=%) \
>>>> + $(AVOCADO_CMDLINE_TAGS) \
>>>> + $(if $(GITLAB_CI),,--failfast)
>>>> $(AVOCADO_TESTS), \
>>>> "AVOCADO", "tests/avocado")
>>>>
>>>> check-acceptance-deprecated-warning:
>>>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>>> index 0ba561b6bd..a6f73da681 100644
>>>> --- a/tests/requirements.txt
>>>> +++ b/tests/requirements.txt
>>>> @@ -2,5 +2,5 @@
>>>> # in the tests/venv Python virtual environment. For more info,
>>>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>> # Note that qemu.git/python/ is always implicitly installed.
>>>> -avocado-framework==88.1
>>>> +avocado-framework==101.0
>>>> pycdlib==1.11.0
>>>> --
>>>> 2.39.2
>>>>
>>>
>>> I thought there were test failures that prohibited us from bumping the
>>> Avocado-Framework version. Did those get rectified recently?
>
> No, still not working on Darwin. At this point I'm thinking at sending
> a patch clarifying testing QEMU with Avocado is not supported on Darwin.
>
Hi Phil,
Do you mean with or without this version bump?
Even though it was somewhat recently[1] that another level of Darwin
compatibility was added to Avocado, where a good part of Avocado's
selftests[2] were enabled in OS X, Avocado 101.0 should not be that far
behind.
I honestly believe people testing and reporting successful execution of
the Avocado tests under Darwin with latest Avocado. Hopefully my memory
is not playing tricks on me, but I'll attempt to verify that.
Cheers,
- Cleber.
[1] - https://github.com/avocado-framework/avocado/pull/5622
[2] -
https://github.com/avocado-framework/avocado/actions/runs/4417285317/jobs/7742772932?pr=5622#step:10:11
>> I'm not seeing any with your patch applied.
>
> Are you using running the Cirrus-CI jobs?
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 16:50 ` John Snow
2023-04-17 17:44 ` Alex Bennée
@ 2023-04-21 3:56 ` Cleber Rosa
1 sibling, 0 replies; 23+ messages in thread
From: Cleber Rosa @ 2023-04-21 3:56 UTC (permalink / raw)
To: John Snow, Alex Bennée, Jan Richter,
Philippe Mathieu-Daudé
Cc: qemu-devel, Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Leif Lindholm, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Kautuk Consul,
Hariharan T S
On 4/17/23 12:50, John Snow wrote:
> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>> tests/Makefile.include | 18 +++++++++++-------
>> tests/requirements.txt | 2 +-
>> 2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>> # download all vm images, according to defined targets
>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>
>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>> +
>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> - $(call quiet-command, \
>> - $(TESTS_PYTHON) -m avocado \
>> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> - --filter-by-tags-include-empty-key) \
>> - $(AVOCADO_CMDLINE_TAGS) \
>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> + $(call quiet-command, \
>> + $(TESTS_PYTHON) -m avocado \
>> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> + $(if $(AVOCADO_TAGS),, \
>> + --filter-by-tags-include-empty \
>> + --filter-by-tags-include-empty-key) \
>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
>> + $(AVOCADO_CMDLINE_TAGS) \
>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> "AVOCADO", "tests/avocado")
>>
>> check-acceptance-deprecated-warning:
>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>> index 0ba561b6bd..a6f73da681 100644
>> --- a/tests/requirements.txt
>> +++ b/tests/requirements.txt
>> @@ -2,5 +2,5 @@
>> # in the tests/venv Python virtual environment. For more info,
>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>> # Note that qemu.git/python/ is always implicitly installed.
>> -avocado-framework==88.1
>> +avocado-framework==101.0
>> pycdlib==1.11.0
>> --
>> 2.39.2
>>
> I thought there were test failures that prohibited us from bumping the
> Avocado-Framework version. Did those get rectified recently?
>
Hi John,
While testing before and after behavior of pretty much every test in
tests/avocado/, I've found that some tests may get affected by the
extra isolation (and overhead) of the new runner in recent Avocado and
may get interrupted more often than on the previous runner. But this is
highly dependent on the amount of resources the machine you run the
tests on has, and how tight the current timeout is.
Running all the tests under old and new Avocado also revealed some tests
that are equally broken under both versions. Thomas has bisected some,
and Cedric has come up with some patches too.
To risk not being too verbose by default, let me know if you want the
full details. Also, see my previous reply about the two issues I'm
aware that would make this bump as seamless as possible.
Thanks,
- Cleber.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 13:43 ` [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0 Alex Bennée
2023-04-17 16:50 ` John Snow
@ 2023-04-18 7:10 ` Thomas Huth
2023-04-21 4:19 ` Cleber Rosa
2023-04-21 3:49 ` Cleber Rosa
2023-04-21 6:24 ` Thomas Huth
3 siblings, 1 reply; 23+ messages in thread
From: Thomas Huth @ 2023-04-18 7:10 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Stefan Hajnoczi, Marc-André Lureau,
Daniel P. Berrangé, Cleber Rosa, Radoslaw Biernacki,
Philippe Mathieu-Daudé, Leif Lindholm, John Snow,
Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Kautuk Consul, Hariharan T S
On 17/04/2023 15.43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
> tests/Makefile.include | 18 +++++++++++-------
> tests/requirements.txt | 2 +-
> 2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> # download all vm images, according to defined targets
> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> - $(call quiet-command, \
> - $(TESTS_PYTHON) -m avocado \
> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> - --filter-by-tags-include-empty-key) \
> - $(AVOCADO_CMDLINE_TAGS) \
> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> + $(call quiet-command, \
> + $(TESTS_PYTHON) -m avocado \
> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> + $(if $(AVOCADO_TAGS),, \
> + --filter-by-tags-include-empty \
> + --filter-by-tags-include-empty-key) \
> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
> + $(AVOCADO_CMDLINE_TAGS) \
> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
We might need to revisit this --failfast logic, too. If I've got that right,
failfast is now the default with the new system? So we might want to disable
it by default again if GITLAB_CI is not set?
Thomas
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-18 7:10 ` Thomas Huth
@ 2023-04-21 4:19 ` Cleber Rosa
2023-04-21 6:21 ` Thomas Huth
0 siblings, 1 reply; 23+ messages in thread
From: Cleber Rosa @ 2023-04-21 4:19 UTC (permalink / raw)
To: Thomas Huth, Alex Bennée, qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Stefan Hajnoczi, Marc-André Lureau,
Daniel P. Berrangé, Radoslaw Biernacki,
Philippe Mathieu-Daudé, Leif Lindholm, John Snow,
Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Kautuk Consul, Hariharan T S
On 4/18/23 03:10, Thomas Huth wrote:
> On 17/04/2023 15.43, Alex Bennée wrote:
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>> tests/Makefile.include | 18 +++++++++++-------
>> tests/requirements.txt | 2 +-
>> 2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>> # download all vm images, according to defined targets
>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%,
>> $(FEDORA_31_DOWNLOAD))
>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
>> -j%,$(MAKEFLAGS))))
>> +
>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> - $(call quiet-command, \
>> - $(TESTS_PYTHON) -m avocado \
>> - --show=$(AVOCADO_SHOW) run
>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> - --filter-by-tags-include-empty-key) \
>> - $(AVOCADO_CMDLINE_TAGS) \
>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> + $(call quiet-command, \
>> + $(TESTS_PYTHON) -m avocado \
>> + --show=$(AVOCADO_SHOW) run
>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>> + $(if $(AVOCADO_TAGS),, \
>> + --filter-by-tags-include-empty \
>> + --filter-by-tags-include-empty-key) \
>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
>> + $(AVOCADO_CMDLINE_TAGS) \
>> + $(if $(GITLAB_CI),,--failfast)
>> $(AVOCADO_TESTS), \
>
> We might need to revisit this --failfast logic, too. If I've got that
> right, failfast is now the default with the new system? So we might
> want to disable it by default again if GITLAB_CI is not set?
>
> Thomas
>
Hi Thomas,
I must be missing something, because under Avocado 101.0, I'm getting
the following behavior without the --failfast flag:
$ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true
JOB ID : 646f476f01b8d5599a57530606de543f2d9a5366
JOB LOG :
/root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log
(1/3) /bin/true: STARTED
(1/3) /bin/true: PASS (0.01 s)
(2/3) /bin/false: STARTED
(2/3) /bin/false: FAIL (0.01 s)
(3/3) /bin/true: STARTED
(3/3) /bin/true: PASS (0.01 s)
RESULTS : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT
0 | CANCEL 0
JOB TIME : 1.01 s
Test summary:
/bin/false: FAIL
And this with --failfast:
$ avocado run --failfast --max-parallel-tasks=1 -- /bin/true
/bin/false /bin/true
JOB ID : ae4894607a42194a7382efa545eccaccf7495fa3
JOB LOG :
/root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log
(1/3) /bin/true: STARTED
(1/3) /bin/true: PASS (0.01 s)
(2/3) /bin/false: STARTED
(2/3) /bin/false: FAIL (0.01 s)
Interrupting job (failfast).
RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT
0 | CANCEL 0
JOB TIME : 0.75 s
Test summary:
/bin/false: FAIL
Maybe it's something in the Makefile I'm missing, or something specific
to these tests, but that would be a long shot.
Thanks,
- Cleber.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-21 4:19 ` Cleber Rosa
@ 2023-04-21 6:21 ` Thomas Huth
0 siblings, 0 replies; 23+ messages in thread
From: Thomas Huth @ 2023-04-21 6:21 UTC (permalink / raw)
To: Cleber Rosa, Alex Bennée, qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Stefan Hajnoczi, Marc-André Lureau,
Daniel P. Berrangé, Radoslaw Biernacki,
Philippe Mathieu-Daudé, Leif Lindholm, John Snow,
Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Kautuk Consul, Hariharan T S
On 21/04/2023 06.19, Cleber Rosa wrote:
>
> On 4/18/23 03:10, Thomas Huth wrote:
>> On 17/04/2023 15.43, Alex Bennée wrote:
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the
>>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due
>>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado
>>> code would start downloading a fresh image from the internet
>>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>> ---
>>> tests/Makefile.include | 18 +++++++++++-------
>>> tests/requirements.txt | 2 +-
>>> 2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>> # download all vm images, according to defined targets
>>> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%,
>>> $(FEDORA_31_DOWNLOAD))
>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>>> +
>>> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> - $(call quiet-command, \
>>> - $(TESTS_PYTHON) -m avocado \
>>> - --show=$(AVOCADO_SHOW) run
>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>> - --filter-by-tags-include-empty-key) \
>>> - $(AVOCADO_CMDLINE_TAGS) \
>>> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> + $(call quiet-command, \
>>> + $(TESTS_PYTHON) -m avocado \
>>> + --show=$(AVOCADO_SHOW) run
>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> + $(if $(AVOCADO_TAGS),, \
>>> + --filter-by-tags-include-empty \
>>> + --filter-by-tags-include-empty-key) \
>>> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
>>> + $(AVOCADO_CMDLINE_TAGS) \
>>> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>
>> We might need to revisit this --failfast logic, too. If I've got that
>> right, failfast is now the default with the new system? So we might want
>> to disable it by default again if GITLAB_CI is not set?
>>
>> Thomas
>>
> Hi Thomas,
>
> I must be missing something, because under Avocado 101.0, I'm getting the
> following behavior without the --failfast flag:
>
> $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true
> JOB ID : 646f476f01b8d5599a57530606de543f2d9a5366
> JOB LOG : /root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log
> (1/3) /bin/true: STARTED
> (1/3) /bin/true: PASS (0.01 s)
> (2/3) /bin/false: STARTED
> (2/3) /bin/false: FAIL (0.01 s)
> (3/3) /bin/true: STARTED
> (3/3) /bin/true: PASS (0.01 s)
> RESULTS : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 |
> CANCEL 0
> JOB TIME : 1.01 s
>
> Test summary:
> /bin/false: FAIL
>
> And this with --failfast:
>
> $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true /bin/false
> /bin/true
> JOB ID : ae4894607a42194a7382efa545eccaccf7495fa3
> JOB LOG : /root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log
> (1/3) /bin/true: STARTED
> (1/3) /bin/true: PASS (0.01 s)
> (2/3) /bin/false: STARTED
> (2/3) /bin/false: FAIL (0.01 s)
> Interrupting job (failfast).
> RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 0 |
> CANCEL 0
> JOB TIME : 0.75 s
>
> Test summary:
> /bin/false: FAIL
>
> Maybe it's something in the Makefile I'm missing, or something specific to
> these tests, but that would be a long shot.
Ah, thanks, I think I simply got it mixed up while trying to read the above
Makefile logic - never mind my question, it should be fine.
Thomas
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 13:43 ` [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0 Alex Bennée
2023-04-17 16:50 ` John Snow
2023-04-18 7:10 ` Thomas Huth
@ 2023-04-21 3:49 ` Cleber Rosa
2023-04-21 6:24 ` Thomas Huth
3 siblings, 0 replies; 23+ messages in thread
From: Cleber Rosa @ 2023-04-21 3:49 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
John Snow, Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Kautuk Consul, Hariharan T S
On 4/17/23 09:43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
> tests/Makefile.include | 18 +++++++++++-------
> tests/requirements.txt | 2 +-
> 2 files changed, 12 insertions(+), 8 deletions(-)
Hi everyone,
Looks like I've mistakenly replied to the earlier thread, so here's
pretty much the same message on a better location.
First of all, thanks to Kautuk for sending this and thanks to Hariharan
for further testing it.
I'd like to give some context which not everyone may be aware of.
Avocado 101.0 is a very different when compared with 88.1. Everything
related to the execution of tests is brand new. To be more precise, on
version 91.0[1], this new runner[2] became the default. On version 97.0,
the old runner implementation (currently in use in QEMU) was finally
removed.
On most releases since then, I've been running the QEMU tests with the
latest Avocado, and finding issues that are (as resources allow)
addressed in later versions. As you probably noticed, Avocado 101.0
runs the QEMU tests without much (or any) visible issues for most
people. But, I'm aware of two pending issues that may or may not be a
big deal to users:
I) The logging behavior is a bit different since Avocado 88.1. At a
given point it was considered that Avocado should not mess around
inadvertently with Python's root logger, and should be more picky about
it includes in logs. For most cases, a simple workaround[4] does the
trick. But, for some other use cases (say for 3rd party libraries' logs
you want logged alongside Avocado's logs) there's a pending PR[5] that
will take care of all known limitations.
II) The support for killing tests (internally in Avocado represented as
more generic "tasks") and all its children is a bit lacking. This is an
issue I'm actively working on[6]. This may leave some processes (such
as "qemu-system-*") running even after a test was interrupted.
Fixes for both of these issues are due to be in version 102.0. The ETA
for version 102.0 is 1-2 weeks.
With that being said, I'm more than OK with this patch (alongside PATCH
2, without which havoc ensues :) provided people understand the two
pending issues above. If this patch is taken before Avocado 102.0 is
released, the delta from 101.0 would be much smaller, so it should be an
easier change to test.
Cheers,
- Cleber.
[1] - https://avocado-framework.readthedocs.io/en/101.0/releases/91_0.html
[2] - The new runner is called "nrunner" and I am to be blamed for the
naming lacking any originality
[3] -
https://avocado-framework.readthedocs.io/en/101.0/releases/97_0.html#users-test-writers
[4] -
https://gitlab.com/cleber.gnu/qemu/-/commit/a9f39c4f6671b756196a185c7275eb7ebd13e588
[5] - https://github.com/avocado-framework/avocado/pull/5645
[6] - https://github.com/avocado-framework/avocado/issues/4994
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0
2023-04-17 13:43 ` [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0 Alex Bennée
` (2 preceding siblings ...)
2023-04-21 3:49 ` Cleber Rosa
@ 2023-04-21 6:24 ` Thomas Huth
3 siblings, 0 replies; 23+ messages in thread
From: Thomas Huth @ 2023-04-21 6:24 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Stefan Hajnoczi, Marc-André Lureau,
Daniel P. Berrangé, Cleber Rosa, Radoslaw Biernacki,
Philippe Mathieu-Daudé, Leif Lindholm, John Snow,
Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Kautuk Consul, Hariharan T S
On 17/04/2023 15.43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
> - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
> tests/Makefile.include | 18 +++++++++++-------
> tests/requirements.txt | 2 +-
> 2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> # download all vm images, according to defined targets
> get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
> check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> - $(call quiet-command, \
> - $(TESTS_PYTHON) -m avocado \
> - --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> - $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> - --filter-by-tags-include-empty-key) \
> - $(AVOCADO_CMDLINE_TAGS) \
> - $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> + $(call quiet-command, \
> + $(TESTS_PYTHON) -m avocado \
> + --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> + $(if $(AVOCADO_TAGS),, \
> + --filter-by-tags-include-empty \
> + --filter-by-tags-include-empty-key) \
> + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \
> + $(AVOCADO_CMDLINE_TAGS) \
> + $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> "AVOCADO", "tests/avocado")
>
> check-acceptance-deprecated-warning:
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 0ba561b6bd..a6f73da681 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,5 +2,5 @@
> # in the tests/venv Python virtual environment. For more info,
> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> # Note that qemu.git/python/ is always implicitly installed.
> -avocado-framework==88.1
> +avocado-framework==101.0
> pycdlib==1.11.0
Reviewed-by: Thomas Huth <thuth@redhat.com>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v4 2/6] tests/avocado: use the new snapshots for testing
2023-04-17 13:43 [PATCH v4 0/6] testing/next: avocado, docs, gitlab Alex Bennée
2023-04-17 13:43 ` [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0 Alex Bennée
@ 2023-04-17 13:43 ` Alex Bennée
2023-04-17 13:43 ` [PATCH v4 3/6] tests/avocado: Add set of boot tests on SBSA-ref Alex Bennée
` (4 subsequent siblings)
6 siblings, 0 replies; 23+ messages in thread
From: Alex Bennée @ 2023-04-17 13:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Alex Bennée, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
John Snow, Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm
The tuxboot images now have a stable snapshot URL so we can enable the
checksums and remove the avocado warnings. We will have to update as
old snapshots retire but that won't be too frequent.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/avocado/tuxrun_baselines.py | 170 +++++++++++++++++++++++++-----
1 file changed, 144 insertions(+), 26 deletions(-)
diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index d343376faa..e8749717e9 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -77,20 +77,33 @@ def wait_for_console_pattern(self, success_message, vm=None):
failure_message='Kernel panic - not syncing',
vm=vm)
- def fetch_tuxrun_assets(self, dt=None):
+ def fetch_tuxrun_assets(self, csums=None, dt=None):
"""
Fetch the TuxBoot assets. They are stored in a standard way so we
use the per-test tags to fetch details.
"""
- base_url = f"https://storage.tuxboot.com/{self.tuxboot}/"
- kernel_image = self.fetch_asset(base_url + self.image)
- disk_image_zst = self.fetch_asset(base_url + "rootfs.ext4.zst")
+ base_url = f"https://storage.tuxboot.com/20230331/{self.tuxboot}/"
+
+ # empty hash if we weren't passed one
+ csums = {} if csums is None else csums
+ ksum = csums.get(self.image, None)
+ isum = csums.get("rootfs.ext4.zst", None)
+
+ kernel_image = self.fetch_asset(base_url + self.image,
+ asset_hash = ksum,
+ algorithm = "sha256")
+ disk_image_zst = self.fetch_asset(base_url + "rootfs.ext4.zst",
+ asset_hash = isum,
+ algorithm = "sha256")
cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
process.run(cmd)
if dt:
- dtb = self.fetch_asset(base_url + dt)
+ dsum = csums.get(dt, None)
+ dtb = self.fetch_asset(base_url + dt,
+ asset_hash = dsum,
+ algorithm = "sha256")
else:
dtb = None
@@ -149,7 +162,9 @@ def run_tuxtest_tests(self, haltmsg):
else:
self.vm.wait()
- def common_tuxrun(self, dt=None,
+ def common_tuxrun(self,
+ csums=None,
+ dt=None,
drive="virtio-blk-device",
haltmsg="reboot: System halted",
console_index=0):
@@ -158,7 +173,7 @@ def common_tuxrun(self, dt=None,
special with the command line we can process most things using
the tag metadata.
"""
- (kernel, disk, dtb) = self.fetch_tuxrun_assets(dt)
+ (kernel, disk, dtb) = self.fetch_tuxrun_assets(csums, dt)
self.prepare_run(kernel, disk, drive, dtb, console_index)
self.vm.launch()
@@ -182,7 +197,11 @@ def test_arm64(self):
:avocado: tags=console:ttyAMA0
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun()
+ sums = {"Image" :
+ "ce95a7101a5fecebe0fe630deee6bd97b32ba41bc8754090e9ad8961ea8674c7",
+ "rootfs.ext4.zst" :
+ "bbd5ed4b9c7d3f4ca19ba71a323a843c6b585e880115df3b7765769dbd9dd061"}
+ self.common_tuxrun(csums=sums)
def test_arm64be(self):
"""
@@ -194,7 +213,11 @@ def test_arm64be(self):
:avocado: tags=console:ttyAMA0
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun()
+ sums = { "Image" :
+ "e0df4425eb2cd9ea9a283e808037f805641c65d8fcecc8f6407d8f4f339561b4",
+ "rootfs.ext4.zst" :
+ "e6ffd8813c8a335bc15728f2835f90539c84be7f8f5f691a8b01451b47fb4bd7"}
+ self.common_tuxrun(csums=sums)
def test_armv5(self):
"""
@@ -206,7 +229,15 @@ def test_armv5(self):
:avocado: tags=console:ttyAMA0
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="virtio-blk-pci",
+ sums = { "rootfs.ext4.zst" :
+ "17177afa74e7294da0642861f08c88ca3c836764299a54bf6d1ce276cb9712a5",
+ "versatile-pb.dtb" :
+ "0bc0c0b0858cefd3c32b385c0d66d97142ded29472a496f4f490e42fc7615b25",
+ "zImage" :
+ "c95af2f27647c12265d75e9df44c22ff5228c59855f54aaa70f41ec2842e3a4d" }
+
+ self.common_tuxrun(csums=sums,
+ drive="virtio-blk-pci",
dt="versatile-pb.dtb")
def test_armv7(self):
@@ -219,7 +250,12 @@ def test_armv7(self):
:avocado: tags=console:ttyAMA0
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun()
+ sums = { "rootfs.ext4.zst" :
+ "ab1fbbeaddda1ffdd45c9405a28cd5370c20f23a7cbc809cc90dc9f243a8eb5a",
+ "zImage" :
+ "4c7a22e9f15875bec06bd2a29d822496571eb297d4f22694099ffcdb19077572" }
+
+ self.common_tuxrun(csums=sums)
def test_armv7be(self):
"""
@@ -232,7 +268,12 @@ def test_armv7be(self):
:avocado: tags=console:ttyAMA0
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun()
+ sums = {"rootfs.ext4.zst" :
+ "42ed46dd2d59986206c5b1f6cf35eab58fe3fd20c96b41aaa16b32f3f90a9835",
+ "zImage" :
+ "7facc62082b57af12015b08f7fdbaf2f123ba07a478367853ae12b219afc9f2f" }
+
+ self.common_tuxrun(csums=sums)
def test_i386(self):
"""
@@ -243,7 +284,12 @@ def test_i386(self):
:avocado: tags=image:bzImage
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="virtio-blk-pci")
+ sums = {"bzImage" :
+ "a3e5b32a354729e65910f5a1ffcda7c14a6c12a55e8213fb86e277f1b76ed956",
+ "rootfs.ext4.zst" :
+ "f15e66b2bf673a210ec2a4b2e744a80530b36289e04f5388aab812b97f69754a" }
+
+ self.common_tuxrun(csums=sums, drive="virtio-blk-pci")
def test_mips32(self):
"""
@@ -256,7 +302,12 @@ def test_mips32(self):
:avocado: tags=root:sda
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+ sums = { "rootfs.ext4.zst" :
+ "fc3da0b4c2f38d74c6d705123bb0f633c76ed953128f9d0859378c328a6d11a0",
+ "vmlinux" :
+ "bfd2172f8b17fb32970ca0c8c58f59c5a4ca38aa5855d920be3a69b5d16e52f0" }
+
+ self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
def test_mips32el(self):
"""
@@ -268,7 +319,12 @@ def test_mips32el(self):
:avocado: tags=root:sda
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+ sums = { "rootfs.ext4.zst" :
+ "e799768e289fd69209c21f4dacffa11baea7543d5db101e8ce27e3bc2c41d90e",
+ "vmlinux" :
+ "8573867c68a8443db8de6d08bb33fb291c189ca2ca671471d3973a3e712096a3" }
+
+ self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
def test_mips64(self):
"""
@@ -280,7 +336,12 @@ def test_mips64(self):
:avocado: tags=root:sda
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+ sums = { "rootfs.ext4.zst" :
+ "69d91eeb04df3d8d172922c6993bb37d4deeb6496def75d8580f6f9de3e431da",
+ "vmlinux" :
+ "09010e51e4b8bcbbd2494786ffb48eca78f228e96e5c5438344b0eac4029dc61" }
+
+ self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
def test_mips64el(self):
"""
@@ -291,7 +352,12 @@ def test_mips64el(self):
:avocado: tags=root:sda
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+ sums = { "rootfs.ext4.zst" :
+ "fba585368f5915b1498ed081863474b2d7ec4e97cdd46d21bdcb2f9698f83de4",
+ "vmlinux" :
+ "d4e08965e2155c4cccce7c5f34d18fe34c636cda2f2c9844387d614950155266" }
+
+ self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
def test_ppc32(self):
"""
@@ -302,7 +368,12 @@ def test_ppc32(self):
:avocado: tags=image:uImage
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="virtio-blk-pci")
+ sums = { "rootfs.ext4.zst" :
+ "8885b9d999cc24d679542a02e9b6aaf48f718f2050ece6b8347074b6ee41dd09",
+ "uImage" :
+ "1a68f74b860fda022fb12e03c5efece8c2b8b590d96cca37a8481a3ae0b3f81f" }
+
+ self.common_tuxrun(csums=sums, drive="virtio-blk-pci")
def test_ppc64(self):
"""
@@ -316,7 +387,12 @@ def test_ppc64(self):
:avocado: tags=extradev:driver=spapr-vscsi
:avocado: tags=root:sda
"""
- self.common_tuxrun(drive="scsi-hd")
+ sums = { "rootfs.ext4.zst" :
+ "1d953e81a4379e537fc8e41e05a0a59d9b453eef97aa03d47866c6c45b00bdff",
+ "vmlinux" :
+ "f22a9b9e924174a4c199f4c7e5d91a2339fcfe51c6eafd0907dc3e09b64ab728" }
+
+ self.common_tuxrun(csums=sums, drive="scsi-hd")
def test_ppc64le(self):
"""
@@ -329,7 +405,12 @@ def test_ppc64le(self):
:avocado: tags=extradev:driver=spapr-vscsi
:avocado: tags=root:sda
"""
- self.common_tuxrun(drive="scsi-hd")
+ sums = { "rootfs.ext4.zst" :
+ "b442678c93fb8abe1f7d3bfa20556488de6b475c22c8fed363f42cf81a0a3906",
+ "vmlinux" :
+ "979eb61b445a010fb13e2b927126991f8ceef9c590fa2be0996c00e293e80cf2" }
+
+ self.common_tuxrun(csums=sums, drive="scsi-hd")
def test_riscv32(self):
"""
@@ -337,7 +418,14 @@ def test_riscv32(self):
:avocado: tags=machine:virt
:avocado: tags=tuxboot:riscv32
"""
- self.common_tuxrun()
+ sums = { "Image" :
+ "89599407d7334de629a40e7ad6503c73670359eb5f5ae9d686353a3d6deccbd5",
+ "fw_jump.elf" :
+ "f2ef28a0b77826f79d085d3e4aa686f1159b315eff9099a37046b18936676985",
+ "rootfs.ext4.zst" :
+ "7168d296d0283238ea73cd5a775b3dd608e55e04c7b92b76ecce31bb13108cba" }
+
+ self.common_tuxrun(csums=sums)
def test_riscv64(self):
"""
@@ -345,7 +433,14 @@ def test_riscv64(self):
:avocado: tags=machine:virt
:avocado: tags=tuxboot:riscv64
"""
- self.common_tuxrun()
+ sums = { "Image" :
+ "cd634badc65e52fb63465ec99e309c0de0369f0841b7d9486f9729e119bac25e",
+ "fw_jump.elf" :
+ "6e3373abcab4305fe151b564a4c71110d833c21f2c0a1753b7935459e36aedcf",
+ "rootfs.ext4.zst" :
+ "b18e3a3bdf27be03da0b285e84cb71bf09eca071c3a087b42884b6982ed679eb" }
+
+ self.common_tuxrun(csums=sums)
def test_s390(self):
"""
@@ -355,7 +450,13 @@ def test_s390(self):
:avocado: tags=image:bzImage
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="virtio-blk-ccw",
+ sums = { "bzImage" :
+ "0414e98dd1c3dafff8496c9cd9c28a5f8d04553bb5ba37e906a812b48d442ef0",
+ "rootfs.ext4.zst" :
+ "88c37c32276677f873a25ab9ec6247895b8e3e6f8259134de2a616080b8ab3fc" }
+
+ self.common_tuxrun(csums=sums,
+ drive="virtio-blk-ccw",
haltmsg="Requesting system halt")
# Note: some segfaults caused by unaligned userspace access
@@ -370,9 +471,14 @@ def test_sh4(self):
:avocado: tags=root:sda
:avocado: tags=console:ttySC1
"""
+ sums = { "rootfs.ext4.zst" :
+ "3592a7a3d5a641e8b9821449e77bc43c9904a56c30d45da0694349cfd86743fd",
+ "zImage" :
+ "29d9b2aba604a0f53a5dc3b5d0f2b8e35d497de1129f8ee5139eb6fdf0db692f" }
+
# The test is currently too unstable to do much in userspace
# so we skip common_tuxrun and do a minimal boot and shutdown.
- (kernel, disk, dtb) = self.fetch_tuxrun_assets()
+ (kernel, disk, dtb) = self.fetch_tuxrun_assets(csums=sums)
# the console comes on the second serial port
self.prepare_run(kernel, disk,
@@ -395,7 +501,13 @@ def test_sparc64(self):
:avocado: tags=root:sda
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+
+ sums = { "rootfs.ext4.zst" :
+ "ad2f1dc436ab51583543d25d2c210cab478645d47078d30d129a66ab0e281d76",
+ "vmlinux" :
+ "e34313e4325ff21deaa3d38a502aa09a373ef62b9bd4d7f8f29388b688225c55" }
+
+ self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
def test_x86_64(self):
"""
@@ -407,4 +519,10 @@ def test_x86_64(self):
:avocado: tags=root:sda
:avocado: tags=shutdown:nowait
"""
- self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+ sums = { "bzImage" :
+ "2bc7480a669ee9b6b82500a236aba0c54233debe98cb968268fa230f52f03461",
+ "rootfs.ext4.zst" :
+ "b72ac729769b8f51c6dffb221113c9a063c774dbe1d66af30eb593c4e9999b4b" }
+
+ self.common_tuxrun(csums=sums,
+ drive="driver=ide-hd,bus=ide.0,unit=0")
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 3/6] tests/avocado: Add set of boot tests on SBSA-ref
2023-04-17 13:43 [PATCH v4 0/6] testing/next: avocado, docs, gitlab Alex Bennée
2023-04-17 13:43 ` [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0 Alex Bennée
2023-04-17 13:43 ` [PATCH v4 2/6] tests/avocado: use the new snapshots for testing Alex Bennée
@ 2023-04-17 13:43 ` Alex Bennée
2023-04-17 13:43 ` [PATCH v4 4/6] gitlab-ci: Avoid to re-run "configure" in the device-crash-test jobs Alex Bennée
` (3 subsequent siblings)
6 siblings, 0 replies; 23+ messages in thread
From: Alex Bennée @ 2023-04-17 13:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Alex Bennée, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
John Snow, Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Marcin Juszkiewicz
From: Philippe Mathieu-Daudé <philmd@linaro.org>
This change adds set of boot tests on SBSA-ref machine:
1. boot firmware up to the EDK2 banner
2. boot Alpine Linux
Prebuilt flash volumes are included, built using upstream documentation.
To unify tests for AArch64/virt and AArch64/sbsa-ref we boot
the same Alpine Linux image on both.
Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230323082813.971535-1-marcin.juszkiewicz@linaro.org>
Reviewed-by: Leif Lindholm <quic_llindhol@quicinc.com>
Message-Id: <20230328171426.14258-1-philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
MAINTAINERS | 1 +
tests/avocado/machine_aarch64_sbsaref.py | 158 +++++++++++++++++++++++
2 files changed, 159 insertions(+)
create mode 100644 tests/avocado/machine_aarch64_sbsaref.py
diff --git a/MAINTAINERS b/MAINTAINERS
index 2c2068ea5c..b31e00f318 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -942,6 +942,7 @@ L: qemu-arm@nongnu.org
S: Maintained
F: hw/arm/sbsa-ref.c
F: docs/system/arm/sbsa.rst
+F: tests/avocado/machine_aarch64_sbsaref.py
Sharp SL-5500 (Collie) PDA
M: Peter Maydell <peter.maydell@linaro.org>
diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
new file mode 100644
index 0000000000..0a79fa7ab6
--- /dev/null
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -0,0 +1,158 @@
+# Functional test that boots a Linux kernel and checks the console
+#
+# SPDX-FileCopyrightText: 2023 Linaro Ltd.
+# SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
+# SPDX-FileContributor: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+
+from avocado import skip
+from avocado import skipUnless
+from avocado.utils import archive
+
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import wait_for_console_pattern
+from avocado_qemu import interrupt_interactive_console_until_pattern
+
+
+class Aarch64SbsarefMachine(QemuSystemTest):
+ """
+ :avocado: tags=arch:aarch64
+ :avocado: tags=machine:sbsa-ref
+ """
+
+ timeout = 180
+
+ def fetch_firmware(self):
+ """
+ Flash volumes generated using:
+
+ - Fedora GNU Toolchain version 12.2.1 20220819 (Red Hat Cross 12.2.1-2)
+
+ - Trusted Firmware-A
+ https://github.com/ARM-software/arm-trusted-firmware/tree/5fdb2e54
+
+ - Tianocore EDK II
+ https://github.com/tianocore/edk2/tree/494127613b
+ https://github.com/tianocore/edk2-non-osi/tree/41876073
+ https://github.com/tianocore/edk2-platforms/tree/8efa4f42
+ """
+
+ # Secure BootRom (TF-A code)
+ fs0_xz_url = (
+ "https://fileserver.linaro.org/s/ATnSmq6k8SoXgbH/"
+ "download/SBSA_FLASH0.fd.xz"
+ )
+ fs0_xz_hash = "a210a09692bcbe0a3743ffd0df44e80e0c7ad8ab"
+ tar_xz_path = self.fetch_asset(fs0_xz_url, asset_hash=fs0_xz_hash)
+ archive.extract(tar_xz_path, self.workdir)
+ fs0_path = os.path.join(self.workdir, "SBSA_FLASH0.fd")
+
+ # Non-secure rom (UEFI and EFI variables)
+ fs1_xz_url = (
+ "https://fileserver.linaro.org/s/t8foNnMPz74DZZy/"
+ "download/SBSA_FLASH1.fd.xz"
+ )
+ fs1_xz_hash = "13a9a262953787c7fc5a9155dfaa26e703631e02"
+ tar_xz_path = self.fetch_asset(fs1_xz_url, asset_hash=fs1_xz_hash)
+ archive.extract(tar_xz_path, self.workdir)
+ fs1_path = os.path.join(self.workdir, "SBSA_FLASH1.fd")
+
+ for path in [fs0_path, fs1_path]:
+ with open(path, "ab+") as fd:
+ fd.truncate(256 << 20) # Expand volumes to 256MiB
+
+ self.vm.set_console()
+ self.vm.add_args(
+ "-drive",
+ f"if=pflash,file={fs0_path},format=raw",
+ "-drive",
+ f"if=pflash,file={fs1_path},format=raw",
+ "-smp",
+ "1",
+ "-machine",
+ "sbsa-ref",
+ )
+
+ def test_sbsaref_edk2_firmware(self):
+ """
+ :avocado: tags=cpu:cortex-a57
+ """
+
+ self.fetch_firmware()
+ self.vm.launch()
+
+ # TF-A boot sequence:
+ #
+ # https://github.com/ARM-software/arm-trusted-firmware/blob/v2.8.0/\
+ # docs/design/trusted-board-boot.rst#trusted-board-boot-sequence
+ # https://trustedfirmware-a.readthedocs.io/en/v2.8/\
+ # design/firmware-design.html#cold-boot
+
+ # AP Trusted ROM
+ wait_for_console_pattern(self, "Booting Trusted Firmware")
+ wait_for_console_pattern(self, "BL1: v2.8(release):v2.8")
+ wait_for_console_pattern(self, "BL1: Booting BL2")
+
+ # Trusted Boot Firmware
+ wait_for_console_pattern(self, "BL2: v2.8(release)")
+ wait_for_console_pattern(self, "Booting BL31")
+
+ # EL3 Runtime Software
+ wait_for_console_pattern(self, "BL31: v2.8(release)")
+
+ # Non-trusted Firmware
+ wait_for_console_pattern(self, "UEFI firmware (version 1.0")
+ interrupt_interactive_console_until_pattern(self, "QEMU SBSA-REF Machine")
+
+ # This tests the whole boot chain from EFI to Userspace
+ # We only boot a whole OS for the current top level CPU and GIC
+ # Other test profiles should use more minimal boots
+ def boot_alpine_linux(self, cpu):
+ self.fetch_firmware()
+
+ iso_url = (
+ "https://dl-cdn.alpinelinux.org/"
+ "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+ )
+
+ iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+ iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
+
+ self.vm.set_console()
+ self.vm.add_args(
+ "-cpu",
+ cpu,
+ "-drive",
+ f"file={iso_path},format=raw",
+ "-device",
+ "virtio-rng-pci,rng=rng0",
+ "-object",
+ "rng-random,id=rng0,filename=/dev/urandom",
+ )
+
+ self.vm.launch()
+ wait_for_console_pattern(self, "Welcome to Alpine Linux 3.17")
+
+ @skipUnless(os.getenv("AVOCADO_TIMEOUT_EXPECTED"), "Test might timeout")
+ def test_sbsaref_alpine_linux_cortex_a57(self):
+ """
+ :avocado: tags=cpu:cortex-a57
+ """
+ self.boot_alpine_linux("cortex-a57")
+
+ @skipUnless(os.getenv("AVOCADO_TIMEOUT_EXPECTED"), "Test might timeout")
+ def test_sbsaref_alpine_linux_neoverse_n1(self):
+ """
+ :avocado: tags=cpu:max
+ """
+ self.boot_alpine_linux("neoverse-n1")
+
+ @skip("requires TF-A update to handle FEAT_FGT")
+ def test_sbsaref_alpine_linux_max(self):
+ """
+ :avocado: tags=cpu:max
+ """
+ self.boot_alpine_linux("max,pauth-impdef=on")
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 4/6] gitlab-ci: Avoid to re-run "configure" in the device-crash-test jobs
2023-04-17 13:43 [PATCH v4 0/6] testing/next: avocado, docs, gitlab Alex Bennée
` (2 preceding siblings ...)
2023-04-17 13:43 ` [PATCH v4 3/6] tests/avocado: Add set of boot tests on SBSA-ref Alex Bennée
@ 2023-04-17 13:43 ` Alex Bennée
2023-04-17 13:43 ` [PATCH v4 5/6] scripts/device-crash-test: Add a parameter to run with TCG only Alex Bennée
` (2 subsequent siblings)
6 siblings, 0 replies; 23+ messages in thread
From: Alex Bennée @ 2023-04-17 13:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Alex Bennée, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
John Snow, Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm
From: Thomas Huth <thuth@redhat.com>
After "make check-venv" had been added to these jobs, they started
to re-run "configure" each time since our logic in the makefile
thinks that some files are out of date here. Avoid it with the same
trick that we are using in buildtest-template.yml already by disabling
the up-to-date check via NINJA=":".
Fixes: 1d8cf47e5b ("tests: run 'device-crash-test' from tests/venv")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230414145845.456145-2-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.gitlab-ci.d/buildtest.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index ba6f551752..333eea9dd3 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -102,7 +102,7 @@ crash-test-debian:
IMAGE: debian-amd64
script:
- cd build
- - make check-venv
+ - make NINJA=":" check-venv
- tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
build-system-fedora:
@@ -145,7 +145,7 @@ crash-test-fedora:
IMAGE: fedora
script:
- cd build
- - make check-venv
+ - make NINJA=":" check-venv
- tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
- tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread* [PATCH v4 5/6] scripts/device-crash-test: Add a parameter to run with TCG only
2023-04-17 13:43 [PATCH v4 0/6] testing/next: avocado, docs, gitlab Alex Bennée
` (3 preceding siblings ...)
2023-04-17 13:43 ` [PATCH v4 4/6] gitlab-ci: Avoid to re-run "configure" in the device-crash-test jobs Alex Bennée
@ 2023-04-17 13:43 ` Alex Bennée
2023-04-20 9:20 ` Philippe Mathieu-Daudé
2023-04-17 13:43 ` [PATCH v4 6/6] qemu-options: finesse the recommendations around -blockdev Alex Bennée
2023-04-17 18:03 ` [PATCH v4 0/6] testing/next: avocado, docs, gitlab John Snow
6 siblings, 1 reply; 23+ messages in thread
From: Alex Bennée @ 2023-04-17 13:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Alex Bennée, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
John Snow, Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Eldon Stegall
From: Thomas Huth <thuth@redhat.com>
We're currently facing the problem that the device-crash-test script
runs twice as long in the CI when a runner supports KVM - which sometimes
results in a timeout of the CI job. To get a more deterministic runtime
here, add an option to the script that allows to run it with TCG only.
Reported-by: Eldon Stegall <eldon-qemu@eldondev.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230414145845.456145-3-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.gitlab-ci.d/buildtest.yml | 2 +-
scripts/device-crash-test | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 333eea9dd3..bb3650a51c 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -103,7 +103,7 @@ crash-test-debian:
script:
- cd build
- make NINJA=":" check-venv
- - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
+ - tests/venv/bin/python3 scripts/device-crash-test -q --tcg-only ./qemu-system-i386
build-system-fedora:
extends:
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 73bcb98693..b74d887331 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -397,7 +397,7 @@ def binariesToTest(args, testcase):
def accelsToTest(args, testcase):
- if getBinaryInfo(args, testcase['binary']).kvm_available:
+ if getBinaryInfo(args, testcase['binary']).kvm_available and not args.tcg_only:
yield 'kvm'
yield 'tcg'
@@ -510,6 +510,8 @@ def main():
help="Full mode: test cases that are expected to fail")
parser.add_argument('--strict', action='store_true', dest='strict',
help="Treat all warnings as fatal")
+ parser.add_argument('--tcg-only', action='store_true', dest='tcg_only',
+ help="Only test with TCG accelerator")
parser.add_argument('qemu', nargs='*', metavar='QEMU',
help='QEMU binary to run')
args = parser.parse_args()
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 5/6] scripts/device-crash-test: Add a parameter to run with TCG only
2023-04-17 13:43 ` [PATCH v4 5/6] scripts/device-crash-test: Add a parameter to run with TCG only Alex Bennée
@ 2023-04-20 9:20 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 23+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-04-20 9:20 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Leif Lindholm, John Snow, Paolo Bonzini,
Wainer dos Santos Moschetta, qemu-arm, Eldon Stegall
On 17/4/23 15:43, Alex Bennée wrote:
> From: Thomas Huth <thuth@redhat.com>
>
> We're currently facing the problem that the device-crash-test script
> runs twice as long in the CI when a runner supports KVM - which sometimes
> results in a timeout of the CI job. To get a more deterministic runtime
> here, add an option to the script that allows to run it with TCG only.
>
> Reported-by: Eldon Stegall <eldon-qemu@eldondev.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> Message-Id: <20230414145845.456145-3-thuth@redhat.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .gitlab-ci.d/buildtest.yml | 2 +-
> scripts/device-crash-test | 4 +++-
> 2 files changed, 4 insertions(+), 2 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v4 6/6] qemu-options: finesse the recommendations around -blockdev
2023-04-17 13:43 [PATCH v4 0/6] testing/next: avocado, docs, gitlab Alex Bennée
` (4 preceding siblings ...)
2023-04-17 13:43 ` [PATCH v4 5/6] scripts/device-crash-test: Add a parameter to run with TCG only Alex Bennée
@ 2023-04-17 13:43 ` Alex Bennée
2023-04-17 18:03 ` [PATCH v4 0/6] testing/next: avocado, docs, gitlab John Snow
6 siblings, 0 replies; 23+ messages in thread
From: Alex Bennée @ 2023-04-17 13:43 UTC (permalink / raw)
To: qemu-devel
Cc: Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Alex Bennée, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
John Snow, Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm,
Michael Tokarev, Kevin Wolf
We are a bit premature in recommending -blockdev/-device as the best
way to configure block devices. It seems there are times the more
human friendly -drive still makes sense especially when -snapshot is
involved.
Improve the language to hopefully make things clearer.
Suggested-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20230330101141.30199-5-alex.bennee@linaro.org>
---
v3
- more re-wording to try and approach consensus
- add explicit warning to -snapshot option
---
qemu-options.hx | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/qemu-options.hx b/qemu-options.hx
index 59bdf67a2c..4b8855a4f7 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -1143,10 +1143,22 @@ have gone through several iterations as the feature set and complexity
of the block layer have grown. Many online guides to QEMU often
reference older and deprecated options, which can lead to confusion.
-The recommended modern way to describe disks is to use a combination of
+The most explicit way to describe disks is to use a combination of
``-device`` to specify the hardware device and ``-blockdev`` to
describe the backend. The device defines what the guest sees and the
-backend describes how QEMU handles the data.
+backend describes how QEMU handles the data. It is the only guaranteed
+stable interface for describing block devices and as such is
+recommended for management tools and scripting.
+
+The ``-drive`` option combines the device and backend into a single
+command line option which is a more human friendly. There is however no
+interface stability guarantee although some older board models still
+need updating to work with the modern blockdev forms.
+
+Older options like ``-hda`` are essentially macros which expand into
+``-drive`` options for various drive interfaces. The original forms
+bake in a lot of assumptions from the days when QEMU was emulating a
+legacy PC, they are not recommended for modern configurations.
ERST
@@ -1639,6 +1651,14 @@ SRST
the raw disk image you use is not written back. You can however
force the write back by pressing C-a s (see the :ref:`disk images`
chapter in the System Emulation Users Guide).
+
+ .. warning::
+ snapshot is incompatible with ``-blockdev`` (instead use qemu-img
+ to manually create snapshot images to attach to your blockdev).
+ If you have mixed ``-blockdev`` and ``-drive`` declarations you
+ can use the 'snapshot' property on your drive declarations
+ instead of this global option.
+
ERST
DEF("fsdev", HAS_ARG, QEMU_OPTION_fsdev,
--
2.39.2
^ permalink raw reply related [flat|nested] 23+ messages in thread* Re: [PATCH v4 0/6] testing/next: avocado, docs, gitlab
2023-04-17 13:43 [PATCH v4 0/6] testing/next: avocado, docs, gitlab Alex Bennée
` (5 preceding siblings ...)
2023-04-17 13:43 ` [PATCH v4 6/6] qemu-options: finesse the recommendations around -blockdev Alex Bennée
@ 2023-04-17 18:03 ` John Snow
6 siblings, 0 replies; 23+ messages in thread
From: John Snow @ 2023-04-17 18:03 UTC (permalink / raw)
To: Alex Bennée
Cc: qemu-devel, Peter Maydell, Bastian Koppelmann, Markus Armbruster,
Beraldo Leal, Thomas Huth, Stefan Hajnoczi,
Marc-André Lureau, Daniel P. Berrangé, Cleber Rosa,
Radoslaw Biernacki, Philippe Mathieu-Daudé, Leif Lindholm,
Paolo Bonzini, Wainer dos Santos Moschetta, qemu-arm
On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> This is the current state of my testing tree. It contains a few
> patches that didn't make it into 8.0 and more testing fixes including
> a bump to the avocado release including Thomas' --max-parallel-tasks
> hack to avoid running the tests in parallel. So far it doesn't look
> like its causing any problems although I am seeing some timeouts on
> the BSDs.
Oh, looks like you answered my question here. OK then!
>
> The following patches need review:
>
> tests/avocado: use the new snapshots for testing
> tests/requirements.txt: bump up avocado-framework version to 101.0 (0 acks, 1 sobs, 1 tbs)
>
> Alex Bennée (2):
> tests/avocado: use the new snapshots for testing
> qemu-options: finesse the recommendations around -blockdev
>
> Kautuk Consul (1):
> tests/requirements.txt: bump up avocado-framework version to 101.0
>
> Philippe Mathieu-Daudé (1):
> tests/avocado: Add set of boot tests on SBSA-ref
>
> Thomas Huth (2):
> gitlab-ci: Avoid to re-run "configure" in the device-crash-test jobs
> scripts/device-crash-test: Add a parameter to run with TCG only
>
> MAINTAINERS | 1 +
> .gitlab-ci.d/buildtest.yml | 6 +-
> qemu-options.hx | 24 +++-
> scripts/device-crash-test | 4 +-
> tests/Makefile.include | 18 ++-
> tests/avocado/machine_aarch64_sbsaref.py | 158 +++++++++++++++++++++
> tests/avocado/tuxrun_baselines.py | 170 +++++++++++++++++++----
> tests/requirements.txt | 2 +-
> 8 files changed, 343 insertions(+), 40 deletions(-)
> create mode 100644 tests/avocado/machine_aarch64_sbsaref.py
>
> --
> 2.39.2
>
^ permalink raw reply [flat|nested] 23+ messages in thread