* [PATCH v5 01/15] gitlab: ensure all meson jobs capture build/meson-logs by default
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:45 ` Thomas Huth
2026-03-04 14:21 ` [PATCH v5 02/15] tests: print reason when I/O test is skipped in TAP mode Daniel P. Berrangé
` (14 subsequent siblings)
15 siblings, 1 reply; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
The build-without-defaults and build-tci jobs do not capture any
artifacts, despite running 'make check'. This has proved a repeated
bug with CI jobs, so introduce a new '.meson_job_template' rule
which always captures 'build/meson-logs'. Jobs can still provide
their own 'artifacts:' config which will override this default
behaviour.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/buildtest-template.yml | 30 ++++++++++++++++++-----------
.gitlab-ci.d/buildtest.yml | 4 +++-
2 files changed, 22 insertions(+), 12 deletions(-)
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index d866cb12bb..005058625e 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,5 +1,20 @@
-.native_build_job_template:
+
+# Any job running meson should capture meson logs
+# by default. Some jobs might override the artifacts
+# to capture further files
+.meson_job_template:
extends: .base_job_template
+ artifacts:
+ name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
+ when: always
+ expire_in: 7 days
+ paths:
+ - build/meson-logs
+ reports:
+ junit: build/meson-logs/*.junit.xml
+
+.native_build_job_template:
+ extends: .meson_job_template
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
cache:
@@ -60,7 +75,7 @@
- build/**/*.c.o.d
.common_test_job_template:
- extends: .base_job_template
+ extends: .meson_job_template
stage: test
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
script:
@@ -87,14 +102,7 @@
# Prevent logs from the build job that run earlier
# from being duplicated in the test job artifacts
- rm -f build/meson-logs/*
- artifacts:
- name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
- when: always
- expire_in: 7 days
- paths:
- - build/meson-logs
- reports:
- junit: build/meson-logs/*.junit.xml
+
.functional_test_job_template:
extends: .common_test_job_template
@@ -125,7 +133,7 @@
QEMU_JOB_FUNCTIONAL: 1
.wasm_build_job_template:
- extends: .base_job_template
+ extends: .meson_job_template
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
before_script:
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 6ad3594522..4b20429c7f 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -652,7 +652,9 @@ build-tci:
# Check our reduced build configurations
build-without-defaults:
- extends: .native_build_job_template
+ extends:
+ - .native_build_job_template
+ - .native_build_artifact_template
needs:
- job: amd64-centos9-container
variables:
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH v5 01/15] gitlab: ensure all meson jobs capture build/meson-logs by default
2026-03-04 14:21 ` [PATCH v5 01/15] gitlab: ensure all meson jobs capture build/meson-logs by default Daniel P. Berrangé
@ 2026-03-04 14:45 ` Thomas Huth
2026-03-04 14:49 ` Daniel P. Berrangé
0 siblings, 1 reply; 19+ messages in thread
From: Thomas Huth @ 2026-03-04 14:45 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Philippe Mathieu-Daudé, qemu-block,
Alex Bennée
On 04/03/2026 15.21, Daniel P. Berrangé wrote:
> The build-without-defaults and build-tci jobs do not capture any
> artifacts, despite running 'make check'. This has proved a repeated
> bug with CI jobs, so introduce a new '.meson_job_template' rule
> which always captures 'build/meson-logs'. Jobs can still provide
> their own 'artifacts:' config which will override this default
> behaviour.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/buildtest-template.yml | 30 ++++++++++++++++++-----------
> .gitlab-ci.d/buildtest.yml | 4 +++-
> 2 files changed, 22 insertions(+), 12 deletions(-)
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index d866cb12bb..005058625e 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -1,5 +1,20 @@
> -.native_build_job_template:
> +
> +# Any job running meson should capture meson logs
> +# by default. Some jobs might override the artifacts
> +# to capture further files
> +.meson_job_template:
> extends: .base_job_template
> + artifacts:
> + name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> + when: always
> + expire_in: 7 days
> + paths:
> + - build/meson-logs
> + reports:
> + junit: build/meson-logs/*.junit.xml
> +
> +.native_build_job_template:
> + extends: .meson_job_template
> stage: build
> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
> cache:
> @@ -60,7 +75,7 @@
> - build/**/*.c.o.d
>
> .common_test_job_template:
> - extends: .base_job_template
> + extends: .meson_job_template
> stage: test
> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
> script:
> @@ -87,14 +102,7 @@
> # Prevent logs from the build job that run earlier
> # from being duplicated in the test job artifacts
> - rm -f build/meson-logs/*
> - artifacts:
> - name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> - when: always
> - expire_in: 7 days
> - paths:
> - - build/meson-logs
> - reports:
> - junit: build/meson-logs/*.junit.xml
> +
>
> .functional_test_job_template:
> extends: .common_test_job_template
> @@ -125,7 +133,7 @@
> QEMU_JOB_FUNCTIONAL: 1
>
> .wasm_build_job_template:
> - extends: .base_job_template
> + extends: .meson_job_template
> stage: build
> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
> before_script:
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 6ad3594522..4b20429c7f 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -652,7 +652,9 @@ build-tci:
>
> # Check our reduced build configurations
> build-without-defaults:
> - extends: .native_build_job_template
> + extends:
> + - .native_build_job_template
> + - .native_build_artifact_template
Why do we need native_build_artifact_template here?
Thomas
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v5 01/15] gitlab: ensure all meson jobs capture build/meson-logs by default
2026-03-04 14:45 ` Thomas Huth
@ 2026-03-04 14:49 ` Daniel P. Berrangé
0 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:49 UTC (permalink / raw)
To: Thomas Huth
Cc: qemu-devel, Pierrick Bouvier, John Snow, Kevin Wolf,
Paolo Bonzini, Cleber Rosa, Hanna Reitz,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée
On Wed, Mar 04, 2026 at 03:45:55PM +0100, Thomas Huth wrote:
> On 04/03/2026 15.21, Daniel P. Berrangé wrote:
> > The build-without-defaults and build-tci jobs do not capture any
> > artifacts, despite running 'make check'. This has proved a repeated
> > bug with CI jobs, so introduce a new '.meson_job_template' rule
> > which always captures 'build/meson-logs'. Jobs can still provide
> > their own 'artifacts:' config which will override this default
> > behaviour.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> > .gitlab-ci.d/buildtest-template.yml | 30 ++++++++++++++++++-----------
> > .gitlab-ci.d/buildtest.yml | 4 +++-
> > 2 files changed, 22 insertions(+), 12 deletions(-)
> >
> > diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> > index d866cb12bb..005058625e 100644
> > --- a/.gitlab-ci.d/buildtest-template.yml
> > +++ b/.gitlab-ci.d/buildtest-template.yml
> > @@ -1,5 +1,20 @@
> > -.native_build_job_template:
> > +
> > +# Any job running meson should capture meson logs
> > +# by default. Some jobs might override the artifacts
> > +# to capture further files
> > +.meson_job_template:
> > extends: .base_job_template
> > + artifacts:
> > + name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> > + when: always
> > + expire_in: 7 days
> > + paths:
> > + - build/meson-logs
> > + reports:
> > + junit: build/meson-logs/*.junit.xml
> > +
> > +.native_build_job_template:
> > + extends: .meson_job_template
> > stage: build
> > image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
> > cache:
> > @@ -60,7 +75,7 @@
> > - build/**/*.c.o.d
> > .common_test_job_template:
> > - extends: .base_job_template
> > + extends: .meson_job_template
> > stage: test
> > image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
> > script:
> > @@ -87,14 +102,7 @@
> > # Prevent logs from the build job that run earlier
> > # from being duplicated in the test job artifacts
> > - rm -f build/meson-logs/*
> > - artifacts:
> > - name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> > - when: always
> > - expire_in: 7 days
> > - paths:
> > - - build/meson-logs
> > - reports:
> > - junit: build/meson-logs/*.junit.xml
> > +
> > .functional_test_job_template:
> > extends: .common_test_job_template
> > @@ -125,7 +133,7 @@
> > QEMU_JOB_FUNCTIONAL: 1
> > .wasm_build_job_template:
> > - extends: .base_job_template
> > + extends: .meson_job_template
> > stage: build
> > image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
> > before_script:
> > diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> > index 6ad3594522..4b20429c7f 100644
> > --- a/.gitlab-ci.d/buildtest.yml
> > +++ b/.gitlab-ci.d/buildtest.yml
> > @@ -652,7 +652,9 @@ build-tci:
> > # Check our reduced build configurations
> > build-without-defaults:
> > - extends: .native_build_job_template
> > + extends:
> > + - .native_build_job_template
> > + - .native_build_artifact_template
>
> Why do we need native_build_artifact_template here?
Sigh, that's a leftover a previous iteration of the patch before I
decided to change the base job template.
With regards,
Daniel
--
|: https://berrange.com ~~ https://hachyderm.io/@berrange :|
|: https://libvirt.org ~~ https://entangle-photo.org :|
|: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v5 02/15] tests: print reason when I/O test is skipped in TAP mode
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 01/15] gitlab: ensure all meson jobs capture build/meson-logs by default Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 03/15] tests: remove redundant meson suite for iotests Daniel P. Berrangé
` (13 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
The TAP output on a skipped test:
ok raw 181 # SKIP
is not informative.
The test program included a reason, and that should be displayed
in TAP mode (it is already shown in non-TAP mode):
ok raw 181 # SKIP Postcopy is not supported
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
tests/qemu-iotests/testrunner.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index e2a3658994..dbe2dddc32 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -174,7 +174,7 @@ def test_print_one_line(self, test: str,
elif status == 'fail':
print(f'not ok {self.env.imgfmt} {test}')
elif status == 'not run':
- print(f'ok {self.env.imgfmt} {test} # SKIP')
+ print(f'ok {self.env.imgfmt} {test} # SKIP {description}')
return
if lasttime:
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 03/15] tests: remove redundant meson suite for iotests
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 01/15] gitlab: ensure all meson jobs capture build/meson-logs by default Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 02/15] tests: print reason when I/O test is skipped in TAP mode Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 04/15] tests: ensure all qcow2 I/O tests are able to be run via make Daniel P. Berrangé
` (12 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
If a test is in the 'block-slow' or 'block-thorough' suite, there is no
need to also add it to the 'slow' or 'thorough' suites.
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
tests/qemu-iotests/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index d7bae71ced..bf588cc2c9 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -33,7 +33,7 @@ foreach format, speed: qemu_iotests_formats
if speed == 'quick'
suites = 'block'
else
- suites = ['block-' + speed, speed]
+ suites = ['block-' + speed]
endif
args = ['-tap', '-' + format]
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 04/15] tests: ensure all qcow2 I/O tests are able to be run via make
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (2 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 03/15] tests: remove redundant meson suite for iotests Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 05/15] scripts/mtest2make: ensure output has stable sorting Daniel P. Berrangé
` (11 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
For block formats marked as 'quick', only tests in the 'auto' group are
added to the meson test suite.
The result of this is that qcow2 tests not in the 'auto' group cannot be
run at all, even if passing SPEED=slow or SPEED=thorough.
To fix this we need todo two passes over the I/O test list. First add
all tests from 'auto' group into the 'block' suite, so they are run by
default. Then on the second pass add any tests which were not in 'auto'
into the 'block-slow' suite, so they get run when SPEED=slow or
SPEED=thorough.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
tests/qemu-iotests/meson.build | 48 ++++++++++++++++++++++++++++++----
1 file changed, 43 insertions(+), 5 deletions(-)
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index bf588cc2c9..1a24d801a3 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -30,16 +30,48 @@ endforeach
qemu_iotests_check_cmd = files('check')
foreach format, speed: qemu_iotests_formats
+ # Formats tagged 'quick' get the subset of tests in the 'auto'
+ # group, run by default with 'make check' / 'make check-block'
+ seen = []
if speed == 'quick'
- suites = 'block'
+ args = ['-tap', '-' + format, '-g', 'auto']
+ suites = ['block']
+
+ rc = run_command(
+ [python, qemu_iotests_check_cmd] + args + ['-n'],
+ check: true,
+ )
+
+ foreach item: rc.stdout().strip().split()
+ seen += item
+ args = [qemu_iotests_check_cmd,
+ '-tap', '-' + format, item,
+ '--source-dir', meson.current_source_dir(),
+ '--build-dir', meson.current_build_dir()]
+ # Some individual tests take as long as 45 seconds
+ # Bump the timeout to 3 minutes for some headroom
+ # on slow machines to minimize spurious failures
+ test('io-' + format + '-' + item,
+ python,
+ args: args,
+ depends: qemu_iotests_binaries,
+ env: qemu_iotests_env,
+ protocol: 'tap',
+ timeout: 180,
+ suite: suites)
+ endforeach
+ endif
+
+ suites = []
+ # Any format tagged quick or slow also gets added to slow
+ # otherwise its tagged thorough
+ if speed != 'thorough'
+ suites += ['block-slow']
else
- suites = ['block-' + speed]
+ suites += ['block-thorough']
endif
args = ['-tap', '-' + format]
- if speed == 'quick'
- args += ['-g', 'auto']
- endif
rc = run_command(
[python, qemu_iotests_check_cmd] + args + ['-n'],
@@ -47,6 +79,12 @@ foreach format, speed: qemu_iotests_formats
)
foreach item: rc.stdout().strip().split()
+ # Skip any tests already added from the 'auto' group
+ # as they're run in the 'quick' suite already
+ if item in seen
+ continue
+ endif
+
args = [qemu_iotests_check_cmd,
'-tap', '-' + format, item,
'--source-dir', meson.current_source_dir(),
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 05/15] scripts/mtest2make: ensure output has stable sorting
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (3 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 04/15] tests: ensure all qcow2 I/O tests are able to be run via make Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 06/15] scripts/mtest2make: support optional tests grouping Daniel P. Berrangé
` (10 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
When debugging mtest2make.py changes it is important to be able to
compare the old and new output. This requires that any lists in the
output have stable sort ordering.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
scripts/mtest2make.py | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 4b252defc3..915f02d600 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -67,8 +67,10 @@ def process_tests(test, targets, suites):
suites[s].speeds.add('thorough')
def emit_prolog(suites, prefix):
- all_targets = ' '.join((f'{prefix}-{k}' for k in suites.keys()))
- all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml' for k in suites.keys()))
+ all_targets = ' '.join((f'{prefix}-{k}'
+ for k in sorted(suites.keys())))
+ all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
+ for k in sorted(suites.keys())))
print()
print(f'all-{prefix}-targets = {all_targets}')
print(f'all-{prefix}-xml = {all_xml}')
@@ -81,12 +83,12 @@ def emit_prolog(suites, prefix):
print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
def emit_suite(name, suite, prefix):
- deps = ' '.join(suite.deps)
+ deps = ' '.join(sorted(suite.deps))
print()
print(f'.{prefix}-{name}.deps = {deps}')
print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
- names = ' '.join(suite.names(name))
+ names = ' '.join(sorted(suite.names(name)))
targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
if not name.endswith('-slow') and not name.endswith('-thorough'):
targets += f' {prefix} {prefix}-report.junit.xml'
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 06/15] scripts/mtest2make: support optional tests grouping
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (4 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 05/15] scripts/mtest2make: ensure output has stable sorting Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 07/15] tests: add a meson suite / make target per block I/O tests format Daniel P. Berrangé
` (9 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
Currently tests can be classified into three speed groups depending on
whether the meson suite name ends in '-slow' or '-thorough' or neither.
This gets turned into make targets that match the name of the meson
suite, with the speed suffix stripped. e.g.
* suite=block -> 'make check-block'
* suite=block-slow -> 'make check-block SPEED=slow'
* suite=block-thorough -> 'make check-block SPEED=thorough'
The set of tests under the "thorough" speed, however, can get rather
large and it would be useful to have a way to expose further make
targets for directly running a particular subset of tests.
This needs a way to run a target without requiring the SPEED variable,
while also not having them enabled by default as if they were 'quick'
tests.
This modifies mtest2make.py to support this idea by allowing for a new
suffix '-optional' on a suite. When this is present, a correspondingly
named make target will be created without the '-optional' suffix which
will never be run automatically.
This is intended to be combined with use of other suites. For example,
a single NBD test might be added to two suites, 'block-thorough' and
'block-nbd-optional'.
This would allow running it as part of all the block tests with
'make check-block SPEED=thorough', and as part of a standalone target
'make check-block-nbd'.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
scripts/mtest2make.py | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 915f02d600..383ea68b16 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -22,7 +22,7 @@ def names(self, base):
print(r'''
SPEED = quick
-.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
+.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
.speed.slow = $(sort $(filter-out %-thorough, $1))
.speed.thorough = $(sort $1)
@@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
s = s[:-9]
suites[s].speeds.add('thorough')
+def target_name(suite):
+ if suite.endswith('-optional'):
+ return suite[0:-9]
+ return suite
+
def emit_prolog(suites, prefix):
- all_targets = ' '.join((f'{prefix}-{k}'
+ all_targets = ' '.join((f'{prefix}-{target_name(k)}'
for k in sorted(suites.keys())))
- all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
+ all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
for k in sorted(suites.keys())))
print()
print(f'all-{prefix}-targets = {all_targets}')
@@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
def emit_suite(name, suite, prefix):
+ tgtname = target_name(name)
deps = ' '.join(sorted(suite.deps))
print()
- print(f'.{prefix}-{name}.deps = {deps}')
- print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
+ print(f'.{prefix}-{tgtname}.deps = {deps}')
+ print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
names = ' '.join(sorted(suite.names(name)))
- targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
- if not name.endswith('-slow') and not name.endswith('-thorough'):
+ targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
+ if not name.endswith('-slow') and \
+ not name.endswith('-thorough') and \
+ not name.endswith('-optional'):
targets += f' {prefix} {prefix}-report.junit.xml'
print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
# for the "base" suite possibly add FOO-slow and FOO-thorough
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 07/15] tests: add a meson suite / make target per block I/O tests format
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (5 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 06/15] scripts/mtest2make: support optional tests grouping Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 08/15] docs/devel/testing: expand documentation for 'make check-block' Daniel P. Berrangé
` (8 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
Currently each block format is classified as either 'quick', 'slow' or
'thorough' and this controls whether its I/O tests are added to the meson
suites 'block-quick', 'block-slow' or 'block-thorough'.
This suites are exposed the 'check-block' make target, accepting the
optional SPEED variable.
As we add more formats to the 'thorough' group, however, it becomes
increasingly large and time consuming to run. What is needed is a make
target that can exercise all tests for an individual format, regardless
of speed classification.
This makes use of the previous enhancement to mtest2make.py to introduce
new meson suites 'block-$FORMAT-optional', which translate to new top
level make targets 'check-block-$FORMAT'. These new targets always run
all tests and as such do not need the SPEED variable to be set, but are
not triggered by 'make check' or 'make check-block'.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
tests/Makefile.include | 3 ++-
tests/qemu-iotests/meson.build | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 59b9a4b922..53f47972a3 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -14,7 +14,8 @@ check-help:
@echo " $(MAKE) check-unit Run qobject tests"
@echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
@echo " $(MAKE) check-tracetool Run tracetool generator tests"
- @echo " $(MAKE) check-block Run block tests"
+ @echo " $(MAKE) check-block Run block tests (all formats)"
+ @echo " $(MAKE) check-block-FORMAT Run block tests (only for FORMAT)"
ifneq ($(filter $(all-check-targets), check-softfloat),)
@echo " $(MAKE) check-tcg Run TCG tests"
@echo " $(MAKE) check-softfloat Run FPU emulation tests"
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 1a24d801a3..16a5e39476 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -62,7 +62,8 @@ foreach format, speed: qemu_iotests_formats
endforeach
endif
- suites = []
+ # Every format gets put in the format specific suite
+ suites = ['block-' + format + '-optional']
# Any format tagged quick or slow also gets added to slow
# otherwise its tagged thorough
if speed != 'thorough'
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 08/15] docs/devel/testing: expand documentation for 'make check-block'
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (6 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 07/15] tests: add a meson suite / make target per block I/O tests format Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 09/15] tests: add nbd and luks to the I/O test suites Daniel P. Berrangé
` (7 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
Explain in greater detail what 'check-block' will run for each format,
and also document the new format specific targets.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
docs/devel/testing/main.rst | 25 ++++++++++++++++++++++---
tests/qemu-iotests/meson.build | 2 ++
2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 0662766b5c..dc4f7202a5 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -221,9 +221,28 @@ same commit that alters the generator code.
check-block
~~~~~~~~~~~
-``make check-block`` runs a subset of the block layer iotests (the tests that
-are in the "auto" group).
-See the "QEMU iotests" section below for more information.
+There are a variety of ways to exercise the block layer I/O tests
+via make targets.
+
+A default ``make check`` or ``make check-block`` command will exercise
+the ``qcow2`` format, using the tests tagged into the ``auto`` group
+only.
+
+These targets accept the ``SPEED`` variable to augment the set of tests
+to run. A slightly more comprehensive test plan can be run by defining
+``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw``
+formats. The most comprehensive test plan can be run by defining
+``SPEED=thorough``, which enables all available tests for the formats
+``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, ``vhdx``,
+``vmdk``, and ``vpc``.
+
+Each of formats also has its own dedicated make target, named
+``make check-block-$FORMAT`` which will run all available tests for
+the designated format and does not require the ``SPEED`` variable
+to be set.
+
+See the "QEMU iotests" section below for more information on the
+block I/O test framework that is leveraged by these ``make`` targets.
.. _qemu-iotests:
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 16a5e39476..66b09d6b97 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -10,6 +10,8 @@ endif
qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
qemu_iotests_env = {'PYTHON': python.full_path()}
+# If altering this definition, also update docs/devel/testing/main.rst
+# section on 'check-block' targets to reflect the changes
qemu_iotests_formats = {
'qcow2': 'quick',
'raw': 'slow',
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 09/15] tests: add nbd and luks to the I/O test suites
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (7 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 08/15] docs/devel/testing: expand documentation for 'make check-block' Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 10/15] tests: use 'driver' as collective term for either format or protocol Daniel P. Berrangé
` (6 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
This introduces new suites for running I/O tests on NBD and LUKS
drivers, giving new make targets
* make check-block-luks
* make check-block-nbd
as well as adding their tests to 'make check-block SPEED=thorough'
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
docs/devel/testing/main.rst | 4 ++--
tests/qemu-iotests/meson.build | 4 +++-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index dc4f7202a5..8067d0a3ed 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -233,8 +233,8 @@ to run. A slightly more comprehensive test plan can be run by defining
``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw``
formats. The most comprehensive test plan can be run by defining
``SPEED=thorough``, which enables all available tests for the formats
-``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, ``vhdx``,
-``vmdk``, and ``vpc``.
+``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``,
+``vhdx``, ``vmdk``, and ``vpc``.
Each of formats also has its own dedicated make target, named
``make check-block-$FORMAT`` which will run all available tests for
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 66b09d6b97..744d0b6e88 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -15,12 +15,14 @@ qemu_iotests_env = {'PYTHON': python.full_path()}
qemu_iotests_formats = {
'qcow2': 'quick',
'raw': 'slow',
+ 'luks': 'thorough',
+ 'nbd': 'thorough',
'parallels': 'thorough',
'qed': 'thorough',
'vdi': 'thorough',
'vhdx': 'thorough',
'vmdk': 'thorough',
- 'vpc': 'thorough'
+ 'vpc': 'thorough',
}
foreach k, v : emulators
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 10/15] tests: use 'driver' as collective term for either format or protocol
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (8 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 09/15] tests: add nbd and luks to the I/O test suites Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 11/15] tests: validate dmsetup result in test 128 Daniel P. Berrangé
` (5 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
The I/O tests integration previously exclusively tested block formats
and now also covers the NBD protocol. Replace references to 'format'
with 'driver', as a generic term to collectively apply to either a
format or protocol.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
docs/devel/testing/main.rst | 13 +++++++------
tests/Makefile.include | 4 ++--
tests/qemu-iotests/meson.build | 6 +++---
3 files changed, 12 insertions(+), 11 deletions(-)
diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 8067d0a3ed..797111009a 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -222,7 +222,8 @@ check-block
~~~~~~~~~~~
There are a variety of ways to exercise the block layer I/O tests
-via make targets.
+via make targets for a selection of formats / protocols (collectively
+referred to as ``drivers`` below).
A default ``make check`` or ``make check-block`` command will exercise
the ``qcow2`` format, using the tests tagged into the ``auto`` group
@@ -231,14 +232,14 @@ only.
These targets accept the ``SPEED`` variable to augment the set of tests
to run. A slightly more comprehensive test plan can be run by defining
``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw``
-formats. The most comprehensive test plan can be run by defining
-``SPEED=thorough``, which enables all available tests for the formats
+drivers. The most comprehensive test plan can be run by defining
+``SPEED=thorough``, which enables all available tests for the drivers
``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``,
``vhdx``, ``vmdk``, and ``vpc``.
-Each of formats also has its own dedicated make target, named
-``make check-block-$FORMAT`` which will run all available tests for
-the designated format and does not require the ``SPEED`` variable
+Each of drivers also has its own dedicated make target, named
+``make check-block-$DRIVER`` which will run all available tests for
+the designated driver and does not require the ``SPEED`` variable
to be set.
See the "QEMU iotests" section below for more information on the
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 53f47972a3..852c01e97b 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -14,8 +14,8 @@ check-help:
@echo " $(MAKE) check-unit Run qobject tests"
@echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
@echo " $(MAKE) check-tracetool Run tracetool generator tests"
- @echo " $(MAKE) check-block Run block tests (all formats)"
- @echo " $(MAKE) check-block-FORMAT Run block tests (only for FORMAT)"
+ @echo " $(MAKE) check-block Run block tests (all formats/protocols)"
+ @echo " $(MAKE) check-block-DRIVER Run block tests (only for format/protocol 'DRIVER')"
ifneq ($(filter $(all-check-targets), check-softfloat),)
@echo " $(MAKE) check-tcg Run TCG tests"
@echo " $(MAKE) check-softfloat Run FPU emulation tests"
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 744d0b6e88..5d6b0b5ed9 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -12,7 +12,7 @@ qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
qemu_iotests_env = {'PYTHON': python.full_path()}
# If altering this definition, also update docs/devel/testing/main.rst
# section on 'check-block' targets to reflect the changes
-qemu_iotests_formats = {
+qemu_iotests_drivers = {
'qcow2': 'quick',
'raw': 'slow',
'luks': 'thorough',
@@ -33,8 +33,8 @@ endforeach
qemu_iotests_check_cmd = files('check')
-foreach format, speed: qemu_iotests_formats
- # Formats tagged 'quick' get the subset of tests in the 'auto'
+foreach format, speed: qemu_iotests_drivers
+ # Drivers tagged 'quick' get the subset of tests in the 'auto'
# group, run by default with 'make check' / 'make check-block'
seen = []
if speed == 'quick'
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 11/15] tests: validate dmsetup result in test 128
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (9 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 10/15] tests: use 'driver' as collective term for either format or protocol Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 12/15] tests: fix check for sudo access in LUKS I/O test Daniel P. Berrangé
` (4 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
The I/O test 128 uses 'dmsetup create' to create a device, optionally
using sudo to elevate privileges.
This dmsetup command works in GitLab CI, however, the test then fails
with a missing device name:
1..1
# running raw 128
not ok raw 128
----------------------------------- stderr -----------------------------------
--- /builds/berrange/qemu/tests/qemu-iotests/128.out
+++ /builds/berrange/qemu/build/scratch/raw-file-128/128.out.bad
@@ -1,5 +1,5 @@
QA output created by 128
== reading from error device ==
-read failed: Input/output error
+qemu-io: can't open device /dev/mapper/eiodev16546: Could not open '/dev/mapper/eiodev16546': No such file or directory
*** done
(test program exited with status code 1)
It is believed that this is due to the build env using a manually
populated /dev, such that the device mapper node won't ever appear.
It is not a race, since a test adding a sleep did not result in the
device appearing.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
tests/qemu-iotests/128 | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/tests/qemu-iotests/128 b/tests/qemu-iotests/128
index d0e00d24b1..d75b1a451a 100755
--- a/tests/qemu-iotests/128
+++ b/tests/qemu-iotests/128
@@ -42,6 +42,12 @@ _setup_eiodev()
echo "0 $((1024 * 1024 * 1024 / 512)) error" | \
$cmd dmsetup create "$devname" 2>/dev/null
if [ "$?" -eq 0 ]; then
+ DEV="/dev/mapper/$devname"
+ if ! -e $DEV
+ then
+ _notrun "Device $DEV not appearing"
+ fi
+
sudo="$cmd"
return
fi
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 12/15] tests: fix check for sudo access in LUKS I/O test
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (10 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 11/15] tests: validate dmsetup result in test 128 Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 13/15] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests Daniel P. Berrangé
` (3 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
The test did not cope with the possibility that 'sudo' was not installed
at all, merely that it was not configured. This broke tests in any CI
env which lacks 'sudo'.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
tests/qemu-iotests/149 | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index c13343d7ef..6dff39a28a 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -95,11 +95,14 @@ def verify_passwordless_sudo():
args = ["sudo", "-n", "/bin/true"]
- proc = subprocess.Popen(args,
- stdin=subprocess.PIPE,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
- universal_newlines=True)
+ try:
+ proc = subprocess.Popen(args,
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ universal_newlines=True)
+ except FileNotFoundError as e:
+ iotests.notrun('requires sudo binary: %s' % e)
msg = proc.communicate()[0]
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 13/15] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (11 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 12/15] tests: fix check for sudo access in LUKS I/O test Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 14/15] gitlab: add jobs for thorough block tests Daniel P. Berrangé
` (2 subsequent siblings)
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
The nature of block I/O tests is such that there can be unexpected false
positive failures in certain scenarios that have not been encountered
before, and sometimes non-deterministic failures that are hard to
reproduce.
Before enabling the I/O tests as gating jobs in CI, there needs to be a
mechanism to dynamically mark tests as skipped, without having to commit
code changes.
This introduces the QEMU_TEST_IO_SKIP environment variable that is set
to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
variable can be set as a GitLab CI pipeline variable to temporarily
disable a test while problems are being debugged.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
docs/devel/testing/main.rst | 7 +++++++
tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 797111009a..f779a64415 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -284,6 +284,13 @@ that are specific to certain cache mode.
More options are supported by the ``./check`` script, run ``./check -h`` for
help.
+If a test program is known to be broken, it can be disabled by setting
+the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
+be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
+leading component can be omitted to skip the test for all formats and
+protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
+will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
+
Writing a new test case
~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index dbe2dddc32..ecb5d4529f 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
self._stack: contextlib.ExitStack
+ self.skip = {}
+ for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
+ rule = rule.strip()
+ if rule == "":
+ continue
+ if ":" in rule:
+ fmt, name = rule.split(":")
+ if fmt in ("", env.imgfmt, env.imgproto):
+ self.skip[name] = True
+ else:
+ self.skip[rule] = True
+
def __enter__(self) -> 'TestRunner':
self._stack = contextlib.ExitStack()
self._stack.enter_context(self.env)
@@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
description='No qualified output '
f'(expected {f_reference})')
+ if f_test.name in self.skip:
+ return TestResult(status='not run',
+ description='Listed in QEMU_TEST_IO_SKIP')
+
args = [str(f_test.resolve())]
env = self.env.prepare_subprocess(args)
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 14/15] gitlab: add jobs for thorough block tests
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (12 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 13/15] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-04 14:21 ` [PATCH v5 15/15] gitlab: remove I/O tests from build-tcg-disabled job Daniel P. Berrangé
2026-03-18 17:02 ` [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
CI is only exercising the qcow2 'auto' tests currently. As a result we
get no exposure of changes which cause regressions in other block format
drivers.
This adds new CI jobs for each block format, that will run the target
'make check-block-$FORMAT'. The jobs are separate so that we have the
ability to make each format gating or not, depending on their level
of reliability.
The 'centos' image is used to run the I/O tests since several tests
have an implicit dependency on x86_64-softmmu, and thus break with
other architecture targets. The 'centos' build job is the only one
that creates the x86_64-softmmu target in CI. Ideally this target
portability in I/O tests would be fixed to avoid this limitation.
There is currently an undiagnosed failure of job 185 with the qcow2
when run in CI that is marked to be skipped, reported at:
https://gitlab.com/qemu-project/qemu/-/issues/3270
The jobs 147 with NBD and 045 with raw are also marked skipped as the
qemu.qmp python code triggers deprecation warnings for sendmsg. This
can be removed when the next release of python-qemu-qmp is made per:
https://gitlab.com/qemu-project/python-qemu-qmp/-/issues/36
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/buildtest.yml | 102 +++++++++++++++++++++++++++++++++++++
1 file changed, 102 insertions(+)
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 4b20429c7f..2e143737a3 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -174,6 +174,108 @@ build-system-centos:
x86_64-softmmu rx-softmmu sh4-softmmu
MAKE_CHECK_ARGS: check-build
+
+# NB: block-XXX jobs use 'centos' since that is the build
+# job that provides the x86_64-softmmu. Some I/O tests
+# are currently buggy and blindly assume characteristics
+# of x86 (such as PCIe) causing failures with other arches
+
+block-luks:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-luks
+
+block-nbd:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-nbd
+ # https://gitlab.com/qemu-project/python-qemu-qmp/-/issues/36
+ QEMU_TEST_IO_SKIP: nbd:147
+
+block-parallels:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-parallels
+
+block-qcow2:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-qcow2
+ # https://gitlab.com/qemu-project/qemu/-/issues/3270
+ QEMU_TEST_IO_SKIP: qcow2:185
+
+block-qed:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-qed
+
+block-raw:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-raw
+ # https://gitlab.com/qemu-project/python-qemu-qmp/-/issues/36
+ QEMU_TEST_IO_SKIP: raw:045
+
+block-vdi:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-vdi
+
+block-vhdx:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-vhdx
+
+block-vmdk:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-vmdk
+
+block-vpc:
+ extends: .native_test_job_template
+ needs:
+ - job: build-system-centos
+ artifacts: true
+ variables:
+ IMAGE: centos9
+ MAKE_CHECK_ARGS: check-block-vpc
+
# Previous QEMU release. Used for cross-version migration tests.
build-previous-qemu:
extends: .native_build_job_template
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH v5 15/15] gitlab: remove I/O tests from build-tcg-disabled job
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (13 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 14/15] gitlab: add jobs for thorough block tests Daniel P. Berrangé
@ 2026-03-04 14:21 ` Daniel P. Berrangé
2026-03-18 17:02 ` [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-04 14:21 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée,
Daniel P. Berrangé
Now that we have dedicated CI jobs for running I/O tests on each
supported format/protocol, we no longer need to special case a
run of a hand picked set of tests in the build-tcg-disabled job.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/buildtest.yml | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 2e143737a3..30fd87c597 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -450,15 +450,6 @@ build-tcg-disabled:
- make -j"$JOBS"
- make check-unit
- make check-qapi-schema
- - ./run tests/qemu-iotests/check -raw 001 002 003 004 005 008 009
- 010 011 012 021 025 032 033 048 052 063 077 086 101 104 106
- 113 148 150 151 152 157 159 160 163 170 171 184 192 194 208
- 221 226 227 236 253 277 image-fleecing
- - ./run tests/qemu-iotests/check -qcow2 028 051 056 057 058 065 068
- 082 085 091 095 096 102 122 124 132 139 142 144 145 151 152
- 155 157 165 194 196 200 202 208 209 216 218 227 234 246 247
- 248 250 254 255 257 258 260 261 262 263 264 270 272 273 277
- 279 image-fleecing
- make distclean
build-user:
--
2.53.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH v5 00/15] tests: do more testing of block drivers
2026-03-04 14:21 [PATCH v5 00/15] tests: do more testing of block drivers Daniel P. Berrangé
` (14 preceding siblings ...)
2026-03-04 14:21 ` [PATCH v5 15/15] gitlab: remove I/O tests from build-tcg-disabled job Daniel P. Berrangé
@ 2026-03-18 17:02 ` Daniel P. Berrangé
15 siblings, 0 replies; 19+ messages in thread
From: Daniel P. Berrangé @ 2026-03-18 17:02 UTC (permalink / raw)
To: qemu-devel
Cc: Pierrick Bouvier, John Snow, Kevin Wolf, Paolo Bonzini,
Cleber Rosa, Hanna Reitz, Thomas Huth,
Philippe Mathieu-Daudé, qemu-block, Alex Bennée
Do the block maintainers have any feedback on this ?
It would nice to get more of the block I/O tests gating merges
before we get regressions sneaking back in.....
On Wed, Mar 04, 2026 at 02:21:05PM +0000, Daniel P. Berrangé wrote:
> v1: https://lists.nongnu.org/archive/html/qemu-devel/2025-10/msg01650.html
> v2: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02083.html
> v3: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02939.html
> v4: https://lists.nongnu.org/archive/html/qemu-devel/2026-02/msg03245.html
>
> The recent (at the time of v1) set of regressions identified in the LUKS
> block driver re-inforced that despite having a hugely useful set of I/O
> tests, our CI coverage is still letting through too many bugs.
>
> The core goals of this series were/are:
>
> * Add LUKS and NBD to the tested formats/protocols
> integrated into "make check-block SPEED=thorough"
>
> * Ensure that all qcow2 tests can be run with "SPEED=slow",
> not staying limited to only the 'auto' group used by the
> 'make check-block' target in its default 'quick' mode
>
> * Add 'make check-block-$FORMAT' to expose a standalone
> target for running all tests for a given format (or
> equivalently a protocol)
>
> * Add GitLab CI jobs for exercising tests for all formats
>
> A sample pipeline for this is
>
> https://gitlab.com/berrange/qemu/-/pipelines/2258731804/
>
> Showing the 'block-FORMAT' jobs which test the 10 block formats/protocols
> currently enabled.
>
> This series fixes a couple of bugs
>
> * Test 128 check for device mapper usage doesn't work
> sufficiently well to detect gitlab's containe env
> can't do dynamic devfs
>
> * Test 185 is reliably failing on gitlab
>
> * Two tests fail when run on CentOS 9 because they trigger
> python deprecation warnings. This will be fixed when the
> python-qemu-qmp module is next released.
>
> Testing of pipelines throughout development of v1-v5 postings does not
> exhibit any non-deterministic failures with the shared runners. Hopefully
> that will apply to QEMU's private runners used forthe staging tree too.
>
> At this point it is a chicken & egg problem though.
>
> If we don't bite the bullet and enable block I/O tests in GitLab CI for
> everyone, we'll never find out if they're truely stable enough to rely
> on.
>
> As a mitigation against instability a patch in this series adds a new
> QEMU_TEST_IO_SKIP env variable that we are able to set as a GitLab CI
> env variable. This lets us skip broken tests dynamically without waiting
> for a git commit to disable them in the buildtest.yml config that we
> use for disabling the 3 known broken tests currently.
>
> Changed in v5:
>
> - Adapt for python-qemu-qmp being spun off, by temp disabling
> tests that are broken by it, that we formerly fixed in-tree.
>
> Changed in v4:
>
> - Remove tests from build-tcg-disabled job
>
> Changed in v3:
>
> - Test whether dmsetup create device appeared or not, instead
> of unconditionally skipping test 128 in GitLab
> - Use 'driver' term instead of 'format' or 'protocol'
> - Fix misc typos & rephrasing
>
> Daniel P. Berrangé (15):
> gitlab: ensure all meson jobs capture build/meson-logs by default
> tests: print reason when I/O test is skipped in TAP mode
> tests: remove redundant meson suite for iotests
> tests: ensure all qcow2 I/O tests are able to be run via make
> scripts/mtest2make: ensure output has stable sorting
> scripts/mtest2make: support optional tests grouping
> tests: add a meson suite / make target per block I/O tests format
> docs/devel/testing: expand documentation for 'make check-block'
> tests: add nbd and luks to the I/O test suites
> tests: use 'driver' as collective term for either format or protocol
> tests: validate dmsetup result in test 128
> tests: fix check for sudo access in LUKS I/O test
> tests: add QEMU_TEST_IO_SKIP for skipping I/O tests
> gitlab: add jobs for thorough block tests
> gitlab: remove I/O tests from build-tcg-disabled job
>
> .gitlab-ci.d/buildtest-template.yml | 30 +++++---
> .gitlab-ci.d/buildtest.yml | 115 +++++++++++++++++++++++++---
> docs/devel/testing/main.rst | 33 +++++++-
> scripts/mtest2make.py | 30 +++++---
> tests/Makefile.include | 3 +-
> tests/qemu-iotests/128 | 6 ++
> tests/qemu-iotests/149 | 13 ++--
> tests/qemu-iotests/meson.build | 59 ++++++++++++--
> tests/qemu-iotests/testrunner.py | 18 ++++-
> 9 files changed, 258 insertions(+), 49 deletions(-)
>
> --
> 2.53.0
>
With regards,
Daniel
--
|: https://berrange.com ~~ https://hachyderm.io/@berrange :|
|: https://libvirt.org ~~ https://entangle-photo.org :|
|: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
^ permalink raw reply [flat|nested] 19+ messages in thread