[PATCH v3 00/14] tests: do more testing of block drivers

["Daniel P. Berrangé" <berrange@redhat.com>]

Original (very broken) v1 posting:

  https://lists.nongnu.org/archive/html/qemu-devel/2025-10/msg01650.html

The recent set of regressions identified in the LUKS block driver
re-inforced that despite having a hugely useful set of I/O tests,
our CI coverage is still letting through too many bugs.

The core goals of this series were/are:

 * Add LUKS and NBD to the tested formats/protocols
   integrated into "make check-block SPEED=thorough"

 * Ensure that all qcow2 tests can be run with "SPEED=slow",
   not staying limited to only the 'auto' group used by the
   'make check-block' target in its default 'quick' mode

 * Add  'make check-block-$FORMAT' to expose a standalone
   target for running all tests for a given format (or
   equivalently a protocol)

 * Add GitLab CI jobs for exercising tests for all formats

A sample pipeline for this is

  https://gitlab.com/berrange/qemu/-/pipelines/2258731804/

Showing the 'block-FORMAT-centos' jobs which test the 10
block formats/protocols currently enabled.

This series fixes a couple of bugs

 * sendmsg generates a deprecation warning on centos

 * Test 128 check for device mapper usage doesn't work
   sufficiently well to detect gitlab's containe env
   can't do dynamic devfs

 * Test 185 is reliably failing on gitlab

I'm unclear if there are other non-deterministic failures that
would only appear if we start running the tests regularly, or
against QEMU's private runners instead of gitlab shared runners.

The ones I've fixed have been consistent across quite a few test
pipelines so far with the shared runners.

At this point it is a chicken & egg problem though.

If we don't bite the bullet and enable block I/O tests in GitLab
we'll never find out if they're stable enough to rely on.

As a mitigation against instability a patch in this series adds
a QEMU_TEST_IO_SKIP env variable that we are able to set as a
GitLab CI env variable. This lets us skip broken tests without
waiting for a git commit to disable them.

Changed in v3:

  - Test whether dmsetup create device appeared or not, instead
    of unconditionally skipping test 128 in GitLab
  - Use 'driver' term instead of 'format' or 'protocol'
  - Fix misc typos & rephrasing

Daniel P. Berrangé (14):
  Revert "python/aqmp: fix send_fd_scm for python 3.6.x"
  tests: print reason when I/O test is skipped in TAP mode
  tests: remove redundant meson suite for iotests
  tests: ensure all qcow2 I/O tests are able to be run via make
  scripts/mtest2make: ensure output has stable sorting
  scripts/mtest2make: support optional tests grouping
  tests: add a meson suite / make target per block I/O tests format
  docs/devel/testing: expand documentation for 'make check-block'
  tests: add nbd and luks to the I/O test suites
  tests: use 'driver' as collective term for either format or protocol
  tests: validate dmsetup result in test 128
  tests: fix check for sudo access in LUKS I/O test
  tests: add QEMU_TEST_IO_SKIP for skipping I/O tests
  gitlab: add jobs for thorough block tests

 .gitlab-ci.d/buildtest.yml       | 92 ++++++++++++++++++++++++++++++++
 docs/devel/testing/main.rst      | 33 ++++++++++--
 python/qemu/qmp/qmp_client.py    |  9 ++--
 scripts/mtest2make.py            | 30 +++++++----
 tests/Makefile.include           |  3 +-
 tests/qemu-iotests/128           |  6 +++
 tests/qemu-iotests/149           | 13 +++--
 tests/qemu-iotests/meson.build   | 59 +++++++++++++++++---
 tests/qemu-iotests/testrunner.py | 18 ++++++-
 9 files changed, 229 insertions(+), 34 deletions(-)

-- 
2.52.0