* Re: [PATCH for-11.1?] qemu-iotest 161: fix race
2026-07-28 16:39 [PATCH for-11.1?] qemu-iotest 161: fix race Michael Tokarev
@ 2026-07-28 17:43 ` Stefan Hajnoczi
2026-07-28 17:53 ` Kevin Wolf
2026-07-29 11:04 ` Daniel P. Berrangé
2 siblings, 0 replies; 8+ messages in thread
From: Stefan Hajnoczi @ 2026-07-28 17:43 UTC (permalink / raw)
To: Michael Tokarev; +Cc: qemu-devel, qemu-block, Kevin Wolf
On Tue, Jul 28, 2026 at 12:40 PM Michael Tokarev <mjt@tls.msk.ru> wrote:
>
> qemu-iotest 161 fails intermittedly due to a race condition:
>
> qemu:block / io-qcow2-161 ERROR 2.08s exit status 1
> ――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
> stderr:
> --- tests/qemu-iotests/161.out
> +++ scratch/qcow2-file-161/161.out.bad
> @@ -34,6 +34,8 @@
> *** Commit and then change an option on the backing file
>
> Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
> +qemu-img: TEST_DIR/t.IMGFMT.base: Failed to get "write" lock
> +Is another process using the image [TEST_DIR/t.IMGFMT.base]?
> Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
> Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
> { 'execute': 'qmp_capabilities' }
>
> This test is run right after a previous test, where we've
> killed qemu process which holds the test image open.
> It looks like when the next test is started (with creating
> the image files), old qemu is still running, so we're racing
> with that qemu, and qemu-img is unable to lock the image file
> properly.
>
> We can add some `sleep $little-amount` here, but it is simpler
> and more reliable to just remove the image files, so that new
> ones are created.
>
> Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/2867
> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH for-11.1?] qemu-iotest 161: fix race
2026-07-28 16:39 [PATCH for-11.1?] qemu-iotest 161: fix race Michael Tokarev
2026-07-28 17:43 ` Stefan Hajnoczi
@ 2026-07-28 17:53 ` Kevin Wolf
2026-07-28 20:36 ` Michael Tokarev
2026-07-29 11:04 ` Daniel P. Berrangé
2 siblings, 1 reply; 8+ messages in thread
From: Kevin Wolf @ 2026-07-28 17:53 UTC (permalink / raw)
To: Michael Tokarev; +Cc: qemu-devel, qemu-block
Am 28.07.2026 um 18:39 hat Michael Tokarev geschrieben:
> qemu-iotest 161 fails intermittedly due to a race condition:
>
> qemu:block / io-qcow2-161 ERROR 2.08s exit status 1
> ――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
> stderr:
> --- tests/qemu-iotests/161.out
> +++ scratch/qcow2-file-161/161.out.bad
> @@ -34,6 +34,8 @@
> *** Commit and then change an option on the backing file
>
> Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
> +qemu-img: TEST_DIR/t.IMGFMT.base: Failed to get "write" lock
> +Is another process using the image [TEST_DIR/t.IMGFMT.base]?
> Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
> Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
> { 'execute': 'qmp_capabilities' }
>
> This test is run right after a previous test, where we've
> killed qemu process which holds the test image open.
> It looks like when the next test is started (with creating
> the image files), old qemu is still running, so we're racing
> with that qemu, and qemu-img is unable to lock the image file
> properly.
>
> We can add some `sleep $little-amount` here, but it is simpler
> and more reliable to just remove the image files, so that new
> ones are created.
>
> Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/2867
> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
>
> diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
> index f25effab936..0b8ba0b19e2 100755
> --- a/tests/qemu-iotests/161
> +++ b/tests/qemu-iotests/161
> @@ -106,6 +106,7 @@ echo
> echo "*** Commit and then change an option on the backing file"
> echo
> # Create the images again
> +rm -f -- "$TEST_IMG.base" "$TEST_IMG.int"
> TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
> TEST_IMG="$TEST_IMG.int" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT
> _make_test_img -b "$TEST_IMG.int" -F $IMGFMT
This can only work for the file protocol, but the test also supports
fuse, which is now broken.
Kevin
161 fail [19:51:58] [19:52:03] 5.3s output mismatch (see /home/kwolf/source/qemu/build-clang/scratch/qcow2-fuse-161/161.out.bad)
--- /home/kwolf/source/qemu/tests/qemu-iotests/161.out
+++ /home/kwolf/source/qemu/build-clang/scratch/qcow2-fuse-161/161.out.bad
@@ -33,6 +33,8 @@
*** Commit and then change an option on the backing file
+rm: cannot remove '/tmp/qemu-iotests-9_5xqdbm/qcow2-fuse-161/fuse-t.qcow2.base': Device or resource busy
+rm: cannot remove '/tmp/qemu-iotests-9_5xqdbm/qcow2-fuse-161/fuse-t.qcow2.int': Device or resource busy
Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
Failures: 161
Failed 1 of 1 iotests
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH for-11.1?] qemu-iotest 161: fix race
2026-07-28 17:53 ` Kevin Wolf
@ 2026-07-28 20:36 ` Michael Tokarev
2026-07-29 9:17 ` Kevin Wolf
0 siblings, 1 reply; 8+ messages in thread
From: Michael Tokarev @ 2026-07-28 20:36 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-devel, qemu-block
On 7/28/26 20:53, Kevin Wolf wrote:
>> diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
>> index f25effab936..0b8ba0b19e2 100755
>> --- a/tests/qemu-iotests/161
>> +++ b/tests/qemu-iotests/161
>> @@ -106,6 +106,7 @@ echo
>> echo "*** Commit and then change an option on the backing file"
>> echo
>> # Create the images again
>> +rm -f -- "$TEST_IMG.base" "$TEST_IMG.int"
>> TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
>> TEST_IMG="$TEST_IMG.int" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT
>> _make_test_img -b "$TEST_IMG.int" -F $IMGFMT
>
> This can only work for the file protocol, but the test also supports
> fuse, which is now broken.
I haven't noticed _rm_test_img function. I wonder maybe we can add
this rm into _make_test_img directly, and fix this for all tests?
_make_test_img already calls _rm_test_img - for fuse protocol exactly.
It can call _rm_test_img unconditionally, I guess.
I'll give it a try tomorrow.
Thanks,
/mjt
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH for-11.1?] qemu-iotest 161: fix race
2026-07-28 20:36 ` Michael Tokarev
@ 2026-07-29 9:17 ` Kevin Wolf
2026-07-29 10:46 ` Michael Tokarev
0 siblings, 1 reply; 8+ messages in thread
From: Kevin Wolf @ 2026-07-29 9:17 UTC (permalink / raw)
To: Michael Tokarev; +Cc: qemu-devel, qemu-block
Am 28.07.2026 um 22:36 hat Michael Tokarev geschrieben:
> On 7/28/26 20:53, Kevin Wolf wrote:
>
> > > diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
> > > index f25effab936..0b8ba0b19e2 100755
> > > --- a/tests/qemu-iotests/161
> > > +++ b/tests/qemu-iotests/161
> > > @@ -106,6 +106,7 @@ echo
> > > echo "*** Commit and then change an option on the backing file"
> > > echo
> > > # Create the images again
> > > +rm -f -- "$TEST_IMG.base" "$TEST_IMG.int"
> > > TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
> > > TEST_IMG="$TEST_IMG.int" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT
> > > _make_test_img -b "$TEST_IMG.int" -F $IMGFMT
> >
> > This can only work for the file protocol, but the test also supports
> > fuse, which is now broken.
>
> I haven't noticed _rm_test_img function. I wonder maybe we can add
> this rm into _make_test_img directly, and fix this for all tests?
>
> _make_test_img already calls _rm_test_img - for fuse protocol exactly.
> It can call _rm_test_img unconditionally, I guess.
Using _rm_test_img should do the trick for 161 because it only supports
file and fuse anyway. Those are the obvious two protocols that work with
_rm_test_img.
Some others might work, too, because they just export a local file that
can be deleted normally (nbd, ssh, nfs). _make_test_img() seems to take
care of restarting the NBD server so that it would then serve the new
image. I suppose SSH and NFS don't need to be notified and would
automatically serve the new image. It's not entirely obvious, but it
looks like it could work.
rbd probably breaks, so we might need to exclude that.
Kevin
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH for-11.1?] qemu-iotest 161: fix race
2026-07-29 9:17 ` Kevin Wolf
@ 2026-07-29 10:46 ` Michael Tokarev
0 siblings, 0 replies; 8+ messages in thread
From: Michael Tokarev @ 2026-07-29 10:46 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-devel, qemu-block
On 7/29/26 12:17, Kevin Wolf wrote:
..
> Using _rm_test_img should do the trick for 161 because it only supports
> file and fuse anyway. Those are the obvious two protocols that work with
> _rm_test_img.
actually it wont. Because among others, _rm_test_img run `qemu-img info`
on the image in question, and this is not working either while it's in use:
$ qemu-img info test.qcow2
qemu-img: Could not open 'test.qcow2': Failed to get shared "write" lock
Is another process using the image [test.qcow2]?
this is exactly the problem we're trying to avoid :)
/mjt
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH for-11.1?] qemu-iotest 161: fix race
2026-07-28 16:39 [PATCH for-11.1?] qemu-iotest 161: fix race Michael Tokarev
2026-07-28 17:43 ` Stefan Hajnoczi
2026-07-28 17:53 ` Kevin Wolf
@ 2026-07-29 11:04 ` Daniel P. Berrangé
2026-07-29 11:18 ` Michael Tokarev
2 siblings, 1 reply; 8+ messages in thread
From: Daniel P. Berrangé @ 2026-07-29 11:04 UTC (permalink / raw)
To: Michael Tokarev; +Cc: qemu-devel, qemu-block
On Tue, Jul 28, 2026 at 07:39:16PM +0300, Michael Tokarev wrote:
> qemu-iotest 161 fails intermittedly due to a race condition:
>
> qemu:block / io-qcow2-161 ERROR 2.08s exit status 1
> ――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――
> stderr:
> --- tests/qemu-iotests/161.out
> +++ scratch/qcow2-file-161/161.out.bad
> @@ -34,6 +34,8 @@
> *** Commit and then change an option on the backing file
>
> Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
> +qemu-img: TEST_DIR/t.IMGFMT.base: Failed to get "write" lock
> +Is another process using the image [TEST_DIR/t.IMGFMT.base]?
> Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
> Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
> { 'execute': 'qmp_capabilities' }
>
> This test is run right after a previous test, where we've
> killed qemu process which holds the test image open.
When you say "previous test" can you clarify the scenario you're
hitting. The tests run with TEST_DIR set to a name that is unique
to the test name + format. eg
qemu.git/build/scratch/raw-file-101
so...
> It looks like when the next test is started (with creating
> the image files), old qemu is still running, so we're racing
> with that qemu, and qemu-img is unable to lock the image file
> properly.
...iiuc this can only happen if the exact same test + format is
re-run ?
Is that what you're doing ?
If so, rather than deleting the image we should do two
things
* Make sure tests clean up after themselves if we find one that
is buggy and leaves stuff running
* Make TEST_DIR more unique by adding the current test hardness
PID to is, so re-runs can never clash
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index dbe2dddc32..9c7f9d733e 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -259,7 +259,7 @@ def do_run_test(self, test: str) -> TestResult:
for d in ['TEST_DIR', 'SOCK_DIR']:
env[d] = os.path.join(
env[d],
- f"{self.env.imgfmt}-{self.env.imgproto}-{f_test.name}")
+ f"{self.env.imgfmt}-{self.env.imgproto}-{f_test.name}-{os.getpid()}")
Path(env[d]).mkdir(parents=True, exist_ok=True)
test_dir = env['TEST_DIR']
With regards,
Daniel
--
|: https://berrange.com ~~ https://hachyderm.io/@berrange :|
|: https://libvirt.org ~~ https://entangle-photo.org :|
|: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH for-11.1?] qemu-iotest 161: fix race
2026-07-29 11:04 ` Daniel P. Berrangé
@ 2026-07-29 11:18 ` Michael Tokarev
0 siblings, 0 replies; 8+ messages in thread
From: Michael Tokarev @ 2026-07-29 11:18 UTC (permalink / raw)
To: Daniel P. Berrangé; +Cc: qemu-devel, qemu-block
On 7/29/26 14:04, Daniel P. Berrangé wrote:
> On Tue, Jul 28, 2026 at 07:39:16PM +0300, Michael Tokarev wrote:
>> This test is run right after a previous test, where we've
>> killed qemu process which holds the test image open.
>
> When you say "previous test" can you clarify the scenario you're
> hitting. The tests run with TEST_DIR set to a name that is unique
> to the test name + format. eg
>
> qemu.git/build/scratch/raw-file-101
It is the same SUBtest in qemu-iotests/161.
/mjt
^ permalink raw reply [flat|nested] 8+ messages in thread