Re: [PATCH v2 2/2] iotests: Add coverage for NBD connect thread start delay

[Eric Blake <eblake@redhat.com>]

On Fri, Oct 16, 2026 at 09:13:57AM -0500, Eric Blake wrote:
> On Fri, Oct 16, 2026 at 10:41:36AM +0200, Guillaume Vincke wrote:
> > Test the connect thread starting condition with reconnect-delay=0 when
> > the server disconnects after a successful handshake but before running
> > operations. Prior to the change to nbd_co_establish_connection(), this
> > test would demonstrate two I/O errors. The tests 264 and 277 covering
> > reconnection do not use reconnect-delay=0 so this case is not exercised.
> > 
> > Add a test that provokes multiple errors in order to put an NBD client
> > in the above-mentioned situation and shows that the first operation
> > starts the connect thread and leads to successful subsequent I/O.
> > 
> > Signed-off-by: Guillaume Vincke <guillaume.vincke@ovhcloud.com>
> > ---
> >  tests/qemu-iotests/tests/nbd-stale-connection | 60 +++++++++++++++++++
> >  .../tests/nbd-stale-connection.out            | 17 ++++++
> >  2 files changed, 77 insertions(+)
> >  create mode 100755 tests/qemu-iotests/tests/nbd-stale-connection
> >  create mode 100644 tests/qemu-iotests/tests/nbd-stale-connection.out
> 
> > +
> > +# estimated duration required by the connect thread to reach a running server
> > +MAX_NBD_CONNECT_DURATION = 1
> > +
> 
> > +
> > +    with qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk):
> > +        iotests.log('=== NBD server up. Wait for the connect thread to park '
> > +                    'the channel ===')
> > +        # in CONNECTING, the operation starts the connect thread
> > +        log(vm.qmp('x-qemu-io', device='nbd0', command='write -P 0x5a 0 64k'))
> > +        # wait for the connect thread to get a new channel and stop
> > +        time.sleep(MAX_NBD_CONNECT_DURATION)
> 
> Timeout-based tests are always risky for false negatives when the CI
> system has more load than the development system.  But comparing this
> version of the patch to v1, it looks like you managed to get much more
> reliable witness points.  I'll be running this through testing on my
> own branch of gitlab several times to see if I can tease out any more
> failures, but hopefully it is ready to use. I'll reply again once I've
> made a decision on whether to include it in my current pull request or
> defer it to my next batch of patches.
> 
> > +
> > +    iotests.log('=== NBD server down. The parked channel is closed ===')
> > +
> > +    with qemu_nbd_popen('-k', nbd_sock, '-f', iotests.imgfmt, disk):
> > +        iotests.log('=== NBD server up. Restart the connect thread ===')
> > +        # operation will determine that the channel got closed and restart the
> > +        # connect thread
> > +        log(vm.qmp('x-qemu-io', device='nbd0', command='write -P 0x5a 0 64k'))
> > +        # Wait for it to succeed
> > +        iotests.log('=== Wait for the connect thread to park the new channel. '
> > +                    'Subsequent operations are successful ===')
> > +        time.sleep(MAX_NBD_CONNECT_DURATION)
> > +        log(vm.qmp('x-qemu-io', device='nbd0', command='write -P 0x5a 0 64k'))

It fared better for qcow2, but luks failed:

https://gitlab.com/ebblake/qemu/-/jobs/17236935977

> ▶ 51/52 luks nbd-stale-connection                                            FAIL          
> 51/52 block-luks-optional+block-thorough - qemu:io-luks-nbd-stale-connection FAIL             3.90s   exit status 1
> >>> RUST_BACKTRACE=1 PYTHON=/builds/ebblake/qemu/build/pyvenv/bin/python3.12 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 LD_LIBRARY_PATH=/builds/ebblake/qemu/build/subprojects/libvfio-user/lib MALLOC_PERTURB_=100 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=1 /builds/ebblake/qemu/build/pyvenv/bin/python3.12 /builds/ebblake/qemu/build/../tests/qemu-iotests/check -tap -luks nbd-stale-connection --source-dir /builds/ebblake/qemu/tests/qemu-iotests --build-dir /builds/ebblake/qemu/build/tests/qemu-iotests
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> WARNING: iotests is being run from outside of the configure-time python virtual environment
> current python: /builds/ebblake/qemu/build/pyvenv/bin/python3.12
> pyvenv python:  /builds/ebblake/qemu/build/pyvenv/bin/python3
> Individual python tests will be executed inside the pyvenv,
> but the test runner will continue to run outside.
> Please use the meson run script:
> 	/builds/ebblake/qemu/build/run /builds/ebblake/qemu/build/../tests/qemu-iotests/check -tap -luks nbd-stale-connection --source-dir /builds/ebblake/qemu/tests/qemu-iotests --build-dir /builds/ebblake/qemu/build/tests/qemu-iotests
> --- /builds/ebblake/qemu/tests/qemu-iotests/tests/nbd-stale-connection.out
> +++ /builds/ebblake/qemu/build/scratch/luks-file-nbd-stale-connection/nbd-stale-connection.out.bad
> @@ -1,17 +1,13 @@
>  Start NBD server
> -=== NBD Device connected ===
> -{"return": {}}
> +qemu-nbd: Failed to blk_new_open '/builds/ebblake/qemu/build/scratch/luks-file-nbd-stale-connection/5998-disk': Parameter 'key-secret' is required for cipher
>  Kill NBD server

and for Parallels:

> https://gitlab.com/ebblake/qemu/-/jobs/17236935979
> 
> --- /builds/ebblake/qemu/tests/qemu-iotests/tests/nbd-stale-connection.out
> +++ /builds/ebblake/qemu/build/scratch/parallels-file-nbd-stale-connection/nbd-stale-connection.out.bad
> @@ -5,11 +5,14 @@
>  === NBD server down. Detect the break and start the connect thread ===
>  {"error": {"class": "GenericError", "desc": "write failed: Input/output error"}}
>  Start NBD server
> +Repairing image was not closed correctly
> +Repairing space leaked at the end of the image 135266304
>  === NBD server up. Wait for the connect thread to park the channel ===
> ...

But those failures are distinct from race conditions causing
unexpected socket status; and may be easier to fix them by just
skipping the test for those formats.

At any rate, it will be easier to exclude this series from my current
pull request, and just pick it up in the next round, once we decide
whether skipping luks and parallels is the best course of action, or
try to make the test even more robust.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org