All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v6 00/16] tests: do more testing of block drivers
@ 2026-05-13 11:59 Daniel P. Berrangé
  2026-05-13 11:59 ` [PATCH v6 01/16] python: bump qemu.qmp to v0.0.6 Daniel P. Berrangé
                   ` (16 more replies)
  0 siblings, 17 replies; 34+ messages in thread
From: Daniel P. Berrangé @ 2026-05-13 11:59 UTC (permalink / raw)
  To: qemu-devel
  Cc: Paolo Bonzini, Pierrick Bouvier, Alex Bennée, qemu-block,
	Kevin Wolf, John Snow, Pierrick Bouvier, Hanna Reitz,
	Philippe Mathieu-Daudé, Cleber Rosa, Thomas Huth,
	Daniel P. Berrangé

 v1: https://lists.nongnu.org/archive/html/qemu-devel/2025-10/msg01650.html
 v2: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02083.html
 v3: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02939.html
 v4: https://lists.nongnu.org/archive/html/qemu-devel/2026-02/msg03245.html
 v5: <...somewhere...mailman is dead again at time of sending...>

The recent (at the time of v1) set of regressions identified in the LUKS
block driver re-inforced that despite having a hugely useful set of I/O
tests, our CI coverage is still letting through too many bugs.

The core goals of this series were/are:

 * Add LUKS and NBD to the tested formats/protocols
   integrated into "make check-block SPEED=thorough"

 * Ensure that all qcow2 tests can be run with "SPEED=slow",
   not staying limited to only the 'auto' group used by the
   'make check-block' target in its default 'quick' mode

 * Add  'make check-block-$FORMAT' to expose a standalone
   target for running all tests for a given format (or
   equivalently a protocol)

 * Add GitLab CI jobs for exercising tests for all formats

A sample pipeline for this is

  https://gitlab.com/berrange/qemu/-/pipelines/2258731804/

Showing the 'block-FORMAT' jobs which test the 10 block formats/protocols
currently enabled.

This series fixes a couple of bugs

 * Test 128 check for device mapper usage doesn't work
   sufficiently well to detect gitlab's containe env
   can't do dynamic devfs

 * Test 185 is failing on gitlab so is marked as flaky

Testing of pipelines throughout development of v1-v5 postings does not
exhibit any non-deterministic failures with the shared runners. Hopefully
that will apply to QEMU's private runners used forthe staging tree too.

At this point it is a chicken & egg problem though.

If we don't bite the bullet and enable block I/O tests in GitLab CI for
everyone, we'll never find out if they're truely stable enough to rely
on.

As a mitigation against instability a patch in this series adds a new
QEMU_TEST_IO_SKIP env variable that we are able to set as a GitLab CI
env variable. This lets us skip broken tests dynamically without waiting
for a git commit to disable them in the buildtest.yml config that we
use for disabling the 3 known broken tests currently.

Changed in v6:

  - Support QEMU_TEST_FLAKY_TESTS in I/O tests framework
    and mark 185 as flaky
  - Drop the env var for skipping tests
  - Use a matrix job in gitlab definition for I/O tests jobs

Changed in v5:

  - Adapt for python-qemu-qmp being spun off, by temp disabling
    tests that are broken by it, that we formerly fixed in-tree.

Changed in v4:

  - Remove tests from build-tcg-disabled job

Changed in v3:

  - Test whether dmsetup create device appeared or not, instead
    of unconditionally skipping test 128 in GitLab
  - Use 'driver' term instead of 'format' or 'protocol'
  - Fix misc typos & rephrasing

Daniel P. Berrangé (15):
  gitlab: ensure all meson jobs capture build/meson-logs by default
  tests: print reason when I/O test is skipped in TAP mode
  tests: remove redundant meson suite for iotests
  tests: ensure all qcow2 I/O tests are able to be run via make
  scripts/mtest2make: ensure output has stable sorting
  scripts/mtest2make: support optional tests grouping
  tests: add a meson suite / make target per block I/O tests format
  docs/devel/testing: expand documentation for 'make check-block'
  tests: add nbd and luks to the I/O test suites
  tests: use 'driver' as collective term for either format or protocol
  tests: validate dmsetup result in test 128
  tests: fix check for sudo access in LUKS I/O test
  tests/qemu-iotests: mark 185 as a flaky test
  gitlab: add jobs for thorough block tests
  gitlab: remove I/O tests from build-tcg-disabled job

John Snow (1):
  python: bump qemu.qmp to v0.0.6

 .gitlab-ci.d/buildtest-template.yml           |  30 +++++----
 .gitlab-ci.d/buildtest.yml                    |  27 +++++---
 docs/devel/testing/main.rst                   |  26 +++++++-
 python/scripts/vendor.py                      |   4 +-
 ...ny.whl => qemu_qmp-0.0.6-py3-none-any.whl} | Bin 72263 -> 72379 bytes
 pythondeps.toml                               |   2 +-
 scripts/mtest2make.py                         |  30 ++++++---
 tests/Makefile.include                        |   3 +-
 tests/qemu-iotests/128                        |   6 ++
 tests/qemu-iotests/149                        |  13 ++--
 tests/qemu-iotests/185                        |   1 +
 tests/qemu-iotests/common.rc                  |  16 +++++
 tests/qemu-iotests/meson.build                |  59 +++++++++++++++---
 tests/qemu-iotests/testrunner.py              |   2 +-
 14 files changed, 168 insertions(+), 51 deletions(-)
 rename python/wheels/{qemu_qmp-0.0.5-py3-none-any.whl => qemu_qmp-0.0.6-py3-none-any.whl} (64%)

-- 
2.54.0



^ permalink raw reply	[flat|nested] 34+ messages in thread
end of thread, other threads:[~2026-05-18 13:13 UTC | newest]

Thread overview: 34+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-05-13 11:59 [PATCH v6 00/16] tests: do more testing of block drivers Daniel P. Berrangé
2026-05-13 11:59 ` [PATCH v6 01/16] python: bump qemu.qmp to v0.0.6 Daniel P. Berrangé
2026-05-13 11:59 ` [PATCH v6 02/16] gitlab: ensure all meson jobs capture build/meson-logs by default Daniel P. Berrangé
2026-05-13 11:59 ` [PATCH v6 03/16] tests: print reason when I/O test is skipped in TAP mode Daniel P. Berrangé
2026-05-15 14:53   ` Kevin Wolf
2026-05-13 11:59 ` [PATCH v6 04/16] tests: remove redundant meson suite for iotests Daniel P. Berrangé
2026-05-13 11:59 ` [PATCH v6 05/16] tests: ensure all qcow2 I/O tests are able to be run via make Daniel P. Berrangé
2026-05-15 14:54   ` Kevin Wolf
2026-05-13 11:59 ` [PATCH v6 06/16] scripts/mtest2make: ensure output has stable sorting Daniel P. Berrangé
2026-05-13 11:59 ` [PATCH v6 07/16] scripts/mtest2make: support optional tests grouping Daniel P. Berrangé
2026-05-13 11:59 ` [PATCH v6 08/16] tests: add a meson suite / make target per block I/O tests format Daniel P. Berrangé
2026-05-15 14:54   ` Kevin Wolf
2026-05-13 11:59 ` [PATCH v6 09/16] docs/devel/testing: expand documentation for 'make check-block' Daniel P. Berrangé
2026-05-15 14:55   ` Kevin Wolf
2026-05-13 11:59 ` [PATCH v6 10/16] tests: add nbd and luks to the I/O test suites Daniel P. Berrangé
2026-05-15 14:55   ` Kevin Wolf
2026-05-13 11:59 ` [PATCH v6 11/16] tests: use 'driver' as collective term for either format or protocol Daniel P. Berrangé
2026-05-15 14:48   ` Kevin Wolf
2026-05-13 11:59 ` [PATCH v6 12/16] tests: validate dmsetup result in test 128 Daniel P. Berrangé
2026-05-13 15:56   ` Pierrick Bouvier
2026-05-15 14:59   ` Kevin Wolf
2026-05-18 13:12     ` Daniel P. Berrangé
2026-05-13 12:00 ` [PATCH v6 13/16] tests: fix check for sudo access in LUKS I/O test Daniel P. Berrangé
2026-05-15 15:01   ` Kevin Wolf
2026-05-13 12:00 ` [PATCH v6 14/16] tests/qemu-iotests: mark 185 as a flaky test Daniel P. Berrangé
2026-05-13 12:14   ` Thomas Huth
2026-05-13 15:59   ` Pierrick Bouvier
2026-05-15 15:05   ` Kevin Wolf
2026-05-13 12:00 ` [PATCH v6 15/16] gitlab: add jobs for thorough block tests Daniel P. Berrangé
2026-05-13 16:00   ` Pierrick Bouvier
2026-05-13 17:18     ` Daniel P. Berrangé
2026-05-13 17:21       ` Pierrick Bouvier
2026-05-13 12:00 ` [PATCH v6 16/16] gitlab: remove I/O tests from build-tcg-disabled job Daniel P. Berrangé
2026-05-15 15:11 ` [PATCH v6 00/16] tests: do more testing of block drivers Kevin Wolf

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.