[PATCH 0/4] ublk: report recovery test failures and fix the exposed defects

[PATCH 0/4] ublk: report recovery test failures and fix the exposed defects

[shao.mingyin]

The four patches make the ublk recovery tests report failures from their
workers, and then fix the two defects that the stricter tests exposed.
They are one series because the later work came out of the earlier
patches: patch 2 applies on top of patch 1, and patches 3 and 4 are the
two independent fixes that patch 2's testing uncovered.

Patch 1 waits for the recorded recovery worker PIDs and reports their
status from the parent, so a failed worker no longer ends in PASS.  It is
a v2 of the patch posted separately on 11-09; the only change is the
simplified wait loop Ming Lei asked for, and his Reviewed-by is kept.

Patch 2 generalizes that collector for the stress scripts, which used a
bare wait in nine of them and could print PASS after a worker failed.

Patch 3 cancels a command that is committed after the quiesce
cancellation scan has passed its slot; without it the server waits for a
completion that never comes and the device stays LIVE.  This is the
kernel fix and carries Fixes: and Cc: stable.

Patch 4 clears the uring_cmd flag word when a selftest batch SQE is
reused, so a recycled slot no longer carries the multishot flag of the
previous batch fetch and is rejected with -EINVAL.

Tested on next-20261105 with CONFIG_BLK_DEV_UBLK=y in a local QEMU/KVM
VM (4 vCPUs, 4 GiB), with real ublk devices, kublk and fio.  Patch 1
keeps its 71-scenario stub matrix.  For patch 2, all nine stress scripts
passed before and after; with an injected ADD_DEV failure all nine
previously reported PASS with exit status 0, and now report FAIL with
255.  For patch 3, a test-only server hook held one COMMIT across a
successful quiesce: the original kernel stayed LIVE until a second
quiesce, the patched kernel reached QUIESCED directly, two runs per
version.  For patch 4, the original kublk rejected the reused batch SQE
with -EINVAL and its daemon exited, the patched kublk kept the device
LIVE.  With both fixes applied the four recovery scripts passed twice
per version.

Shao Mingyin (4):
  selftests: ublk: propagate recovery worker failures
  selftests: ublk: propagate stress worker failures
  ublk: cancel commits arriving after the quiesce scan
  selftests: ublk: clear reused batch SQE command flags

 drivers/block/ublk_drv.c                        | 40 ++++++++++++++++---------
 tools/testing/selftests/ublk/batch.c            |  1 +
 tools/testing/selftests/ublk/test_common.sh     | 30 ++++++++++++++-----
 tools/testing/selftests/ublk/test_recover_01.sh | 20 ++++++++++---
 tools/testing/selftests/ublk/test_recover_02.sh | 20 ++++++++++---
 tools/testing/selftests/ublk/test_recover_03.sh | 20 ++++++++++---
 tools/testing/selftests/ublk/test_recover_04.sh | 14 +++++++--
 tools/testing/selftests/ublk/test_stress_01.sh  |  9 ++++--
 tools/testing/selftests/ublk/test_stress_02.sh  |  9 ++++--
 tools/testing/selftests/ublk/test_stress_03.sh  | 23 +++++++++++---
 tools/testing/selftests/ublk/test_stress_04.sh  | 23 +++++++++++---
 tools/testing/selftests/ublk/test_stress_05.sh  | 28 ++++++++++++-----
 tools/testing/selftests/ublk/test_stress_06.sh  | 15 ++++++++--
 tools/testing/selftests/ublk/test_stress_07.sh  | 15 ++++++++--
 tools/testing/selftests/ublk/test_stress_08.sh  | 10 +++++--
 tools/testing/selftests/ublk/test_stress_09.sh  | 10 +++++--
 16 files changed, 221 insertions(+), 66 deletions(-)

---
base-commit: ddfe51b0a0f5dc9e415cc2c8ad8db17532dd426b

[PATCH 1/4] selftests: ublk: propagate recovery worker failures

[shao.mingyin]

From: Shao Mingyin <shao.mingyin@zte.com.cn>

The recovery tests update ERR_CODE in background subshells and use wait
without PIDs. A worker can report a failure while the parent still prints
PASS and exits with zero. Calling _show_result() from a failed worker also
removes shared backing files while other workers may still be using them.

Wait for each recorded worker PID and aggregate its status in the parent,
keeping failures ahead of skips and preserving errors across batches.
Return worker and device-creation errors without reporting or cleaning up
from the child, so the parent can finish waiting before shared cleanup.

Fixes: 57e13a2e8cd2 ("selftests: ublk: support user recovery")
Fixes: 533c87e2ed74 ("selftests: ublk: add test for UBLK_F_QUIESCE")
Fixes: 63276182c513 ("selftests: ublk: add user copy test cases")
Reviewed-by: Ming Lei <tom.leiming@gmail.com>
Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
---
v2:
  - simplify the worker wait loop, as requested by Ming Lei.
  - the simplified loop is exercised by the real-kernel recovery runs
    described in the stress-worker patch of this series.
v1: https://lore.kernel.org/all/202611092358134332uegNg75loROSwL53Em6A@zte.com.cn/

Tested on native x86_64 Bash 5.2 with the recovery scripts and common
helpers from the unpatched and patched trees. Device and I/O operations
were replaced by test-only stubs. Single-worker failures at every launch
position, multiple failures, skips, signal termination and device-creation
errors no longer produce a final PASS or a zero exit status. Successful
runs still pass, and cleanup follows all worker exits. Prerequisite skips
still exit before preparation. The 71 scenarios passed their expected
checks on each tree. This does not exercise real ublk devices or I/O.

 tools/testing/selftests/ublk/test_common.sh   | 20 +++++++++++++++++--
 .../testing/selftests/ublk/test_recover_01.sh | 20 +++++++++++++++----
 .../testing/selftests/ublk/test_recover_02.sh | 20 +++++++++++++++----
 .../testing/selftests/ublk/test_recover_03.sh | 20 +++++++++++++++----
 .../testing/selftests/ublk/test_recover_04.sh | 14 ++++++++++---
 5 files changed, 77 insertions(+), 17 deletions(-)

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh
index af2ea4f..367df2d 100755
--- a/tools/testing/selftests/ublk/test_common.sh
+++ b/tools/testing/selftests/ublk/test_common.sh
@@ -379,6 +379,23 @@ run_io_and_kill_daemon()
 	fi
 }
 
+# Collect recovery workers in the parent before removing shared backing files.
+_wait_recovery_tests()
+{
+	local pid
+	local ret
+
+	for pid in "$@"; do
+		wait "$pid" && continue
+		ret=$?
+
+		# A failure takes precedence over a skipped worker.
+		if [ "$ERR_CODE" -eq 0 ] || [ "$ERR_CODE" -eq "$UBLK_SKIP_CODE" ]; then
+			ERR_CODE=$ret
+		fi
+	done
+}
+
 run_io_and_recover()
 {
 	local size=$1
@@ -387,8 +404,7 @@ run_io_and_recover()
 	local dev_id
 
 	shift 2
-	dev_id=$(_add_ublk_dev "$@")
-	_check_add_dev "$TID" $?
+	dev_id=$(_add_ublk_dev "$@") || return $?
 
 	fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \
 		--rw=randread --iodepth=256 --size="${size}" --numjobs=4 \
diff --git a/tools/testing/selftests/ublk/test_recover_01.sh b/tools/testing/selftests/ublk/test_recover_01.sh
index 1cddc23..950928a 100755
--- a/tools/testing/selftests/ublk/test_recover_01.sh
+++ b/tools/testing/selftests/ublk/test_recover_01.sh
@@ -11,8 +11,8 @@ ublk_run_recover_test()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -25,20 +25,32 @@ _create_backfile 0 256M
 _create_backfile 1 128M
 _create_backfile 2 128M
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -b &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 -b "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -i 1 &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 -i 1 "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_recover_02.sh b/tools/testing/selftests/ublk/test_recover_02.sh
index 9c3f481..beeeb35 100755
--- a/tools/testing/selftests/ublk/test_recover_02.sh
+++ b/tools/testing/selftests/ublk/test_recover_02.sh
@@ -11,8 +11,8 @@ ublk_run_recover_test()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -29,20 +29,32 @@ _create_backfile 0 256M
 _create_backfile 1 128M
 _create_backfile 2 128M
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -z -b &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 -z -b "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -z -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -z &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 -z "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -z -i 1 &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 -z -i 1 "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -z -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_recover_03.sh b/tools/testing/selftests/ublk/test_recover_03.sh
index 92f4012..0c6b312 100755
--- a/tools/testing/selftests/ublk/test_recover_03.sh
+++ b/tools/testing/selftests/ublk/test_recover_03.sh
@@ -11,8 +11,8 @@ ublk_run_quiesce_recover()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_feature "QUIESCE"; then
@@ -29,20 +29,32 @@ _create_backfile 0 256M
 _create_backfile 1 128M
 _create_backfile 2 128M
 
+pids=()
 ublk_run_quiesce_recover -t null -q 2 -r 1 -b &
+pids+=("$!")
 ublk_run_quiesce_recover -t loop -q 2 -r 1 -b "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_quiesce_recover -t stripe -q 2 -r 1 -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
+pids=()
 ublk_run_quiesce_recover -t null -q 2 -r 1 &
+pids+=("$!")
 ublk_run_quiesce_recover -t loop -q 2 -r 1 "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_quiesce_recover -t stripe -q 2 -r 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
+pids=()
 ublk_run_quiesce_recover -t null -q 2 -r 1 -i 1 &
+pids+=("$!")
 ublk_run_quiesce_recover -t loop -q 2 -r 1 -i 1 "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_quiesce_recover -t stripe -q 2 -r 1 -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_recover_04.sh b/tools/testing/selftests/ublk/test_recover_04.sh
index 4c83c18..232ae7d 100755
--- a/tools/testing/selftests/ublk/test_recover_04.sh
+++ b/tools/testing/selftests/ublk/test_recover_04.sh
@@ -11,8 +11,8 @@ ublk_run_recover_test()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -25,15 +25,23 @@ _create_backfile 0 256M
 _create_backfile 1 128M
 _create_backfile 2 128M
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -u &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 -u "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -u "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
+pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -u -i 1 &
+pids+=("$!")
 ublk_run_recover_test -t loop -q 2 -r 1 -u -i 1 "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -u -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_recovery_tests "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE

[PATCH 2/4] selftests: ublk: propagate stress worker failures

[shao.mingyin]

From: Shao Mingyin <shao.mingyin@zte.com.cn>

The stress tests update ERR_CODE in background subshells and use a bare
wait in the parent. A worker can therefore fail while the parent still
prints PASS and exits successfully. Calling _show_result() from a worker
also removes backing files shared with other workers.

Generalize the recovery PID collector and use it to wait for every stress
worker in the parent. Return errors from the worker helpers, leave result
reporting and cleanup to the parent, and reset the PID array after each
batch. Keep failures ahead of skips when collecting worker exit statuses.

Fixes: bb2cabf23568 ("selftests: ublk: run stress tests in parallel")
Reported-by: Ming Lei <tom.leiming@gmail.com>
Link: https://lore.kernel.org/all/avJ0Vn9G7XhZv-XL@fedora-laptop/
Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
---
This follows "selftests: ublk: propagate recovery worker failures":
https://lore.kernel.org/all/202611092358134332uegNg75loROSwL53Em6A@zte.com.cn/

Tested on next-20261105 with CONFIG_BLK_DEV_UBLK=y in a local QEMU/KVM
VM (4 vCPUs, 4 GiB), using the in-tree kublk and real fio I/O. All nine
stress scripts pass normally before and after the change. A test-only
argument wrapper sets loop ADD_DEV queue counts to zero, making the real
kernel reject them with EINVAL: all nine previously ended with PASS and
exit status 0, and all nine now end with FAIL and exit status 255.

Initial recovery runs exposed failures in recover_01 (remaining QUIESCED)
and recover_03 (remaining LIVE during quiesce). Follow-up experiments
reproduced two independent defects: reused batch SQEs retaining MULTISHOT
and commits arriving after the quiesce cancellation scan.

With separate fixes for those defects applied, the four original
recovery scripts passed twice (eight invocations), without C or driver
diagnostic instrumentation or reduced fio load. This recovery result
includes both separate fixes; the stress A/B results above did not
require them. The original recovery failures are retained in the record.
 tools/testing/selftests/ublk/test_common.sh     | 14 ++++++-------
 tools/testing/selftests/ublk/test_recover_01.sh |  6 +++---
 tools/testing/selftests/ublk/test_recover_02.sh |  6 +++---
 tools/testing/selftests/ublk/test_recover_03.sh |  6 +++---
 tools/testing/selftests/ublk/test_recover_04.sh |  4 ++--
 tools/testing/selftests/ublk/test_stress_01.sh  |  9 ++++++--
 tools/testing/selftests/ublk/test_stress_02.sh  |  9 ++++++--
 tools/testing/selftests/ublk/test_stress_03.sh  | 23 ++++++++++++++++----
 tools/testing/selftests/ublk/test_stress_04.sh  | 23 ++++++++++++++++----
 tools/testing/selftests/ublk/test_stress_05.sh  | 28 ++++++++++++++++++-------
 tools/testing/selftests/ublk/test_stress_06.sh  | 15 ++++++++++---
 tools/testing/selftests/ublk/test_stress_07.sh  | 15 ++++++++++---
 tools/testing/selftests/ublk/test_stress_08.sh  | 10 +++++++--
 tools/testing/selftests/ublk/test_stress_09.sh  | 10 +++++++--
 14 files changed, 130 insertions(+), 48 deletions(-)

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh
index 367df2d..be99f46 100755
--- a/tools/testing/selftests/ublk/test_common.sh
+++ b/tools/testing/selftests/ublk/test_common.sh
@@ -353,13 +353,12 @@ run_io_and_remove()
 	local dev_id
 	shift 1
 
-	dev_id=$(_add_ublk_dev "$@")
-	_check_add_dev "$TID" $?
+	dev_id=$(_add_ublk_dev "$@") || return $?
 
 	[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)"
 	if ! __run_io_and_remove "$dev_id" "${size}" "no"; then
 		echo "/dev/ublkc$dev_id isn't removed"
-		exit 255
+		return 255
 	fi
 }
 
@@ -369,18 +368,17 @@ run_io_and_kill_daemon()
 	local dev_id
 	shift 1
 
-	dev_id=$(_add_ublk_dev "$@")
-	_check_add_dev "$TID" $?
+	dev_id=$(_add_ublk_dev "$@") || return $?
 
 	[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs kill ublk server(ublk add $*)"
 	if ! __run_io_and_remove "$dev_id" "${size}" "yes"; then
 		echo "/dev/ublkc$dev_id isn't removed res ${res}"
-		exit 255
+		return 255
 	fi
 }
 
-# Collect recovery workers in the parent before removing shared backing files.
-_wait_recovery_tests()
+# Collect test workers in the parent before removing shared backing files.
+_wait_test_pids()
 {
 	local pid
 	local ret
diff --git a/tools/testing/selftests/ublk/test_recover_01.sh b/tools/testing/selftests/ublk/test_recover_01.sh
index 950928a..ad5f915 100755
--- a/tools/testing/selftests/ublk/test_recover_01.sh
+++ b/tools/testing/selftests/ublk/test_recover_01.sh
@@ -32,7 +32,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 -b "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 pids=()
 ublk_run_recover_test -t null -q 2 -r 1 &
@@ -41,7 +41,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -i 1 &
@@ -50,7 +50,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 -i 1 "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_recover_02.sh b/tools/testing/selftests/ublk/test_recover_02.sh
index beeeb35..abdc8c1 100755
--- a/tools/testing/selftests/ublk/test_recover_02.sh
+++ b/tools/testing/selftests/ublk/test_recover_02.sh
@@ -36,7 +36,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 -z -b "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -z -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -z &
@@ -45,7 +45,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 -z "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -z -i 1 &
@@ -54,7 +54,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 -z -i 1 "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -z -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_recover_03.sh b/tools/testing/selftests/ublk/test_recover_03.sh
index 0c6b312..aa84354 100755
--- a/tools/testing/selftests/ublk/test_recover_03.sh
+++ b/tools/testing/selftests/ublk/test_recover_03.sh
@@ -36,7 +36,7 @@ ublk_run_quiesce_recover -t loop -q 2 -r 1 -b "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_quiesce_recover -t stripe -q 2 -r 1 -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 pids=()
 ublk_run_quiesce_recover -t null -q 2 -r 1 &
@@ -45,7 +45,7 @@ ublk_run_quiesce_recover -t loop -q 2 -r 1 "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_quiesce_recover -t stripe -q 2 -r 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 pids=()
 ublk_run_quiesce_recover -t null -q 2 -r 1 -i 1 &
@@ -54,7 +54,7 @@ ublk_run_quiesce_recover -t loop -q 2 -r 1 -i 1 "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_quiesce_recover -t stripe -q 2 -r 1 -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_recover_04.sh b/tools/testing/selftests/ublk/test_recover_04.sh
index 232ae7d..ed508b8 100755
--- a/tools/testing/selftests/ublk/test_recover_04.sh
+++ b/tools/testing/selftests/ublk/test_recover_04.sh
@@ -32,7 +32,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 -u "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -u "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 pids=()
 ublk_run_recover_test -t null -q 2 -r 1 -u -i 1 &
@@ -41,7 +41,7 @@ ublk_run_recover_test -t loop -q 2 -r 1 -u -i 1 "${UBLK_BACKFILES[0]}" &
 pids+=("$!")
 ublk_run_recover_test -t stripe -q 2 -r 1 -u -i 1 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 pids+=("$!")
-_wait_recovery_tests "${pids[@]}"
+_wait_test_pids "${pids[@]}"
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_01.sh b/tools/testing/selftests/ublk/test_stress_01.sh
index f91783f..bff70c2 100755
--- a/tools/testing/selftests/ublk/test_stress_01.sh
+++ b/tools/testing/selftests/ublk/test_stress_01.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 ublk_io_and_remove()
 {
@@ -10,8 +11,8 @@ ublk_io_and_remove()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -25,9 +26,13 @@ _create_backfile 1 128M
 _create_backfile 2 128M
 
 ublk_io_and_remove 8G -t null -q 4 &
+pids+=("$!")
 ublk_io_and_remove 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_remove 256M -t stripe -q 4 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_02.sh b/tools/testing/selftests/ublk/test_stress_02.sh
index b128d11..ef3a36f 100755
--- a/tools/testing/selftests/ublk/test_stress_02.sh
+++ b/tools/testing/selftests/ublk/test_stress_02.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 if ! _have_program fio; then
 	exit "$UBLK_SKIP_CODE"
@@ -14,8 +15,8 @@ ublk_io_and_kill_daemon()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 _prep_test "stress" "run IO and kill ublk server"
@@ -26,9 +27,13 @@ _create_backfile 2 128M
 
 for nr_queue in 1 4; do
 	ublk_io_and_kill_daemon 8G -t null -q "$nr_queue" &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 256M -t loop -q "$nr_queue" "${UBLK_BACKFILES[0]}" &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 256M -t stripe -q "$nr_queue" "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-	wait
+	pids+=("$!")
+	_wait_test_pids "${pids[@]}"
+	pids=()
 done
 
 _cleanup_test
diff --git a/tools/testing/selftests/ublk/test_stress_03.sh b/tools/testing/selftests/ublk/test_stress_03.sh
index a0f0aba..9af68c3 100755
--- a/tools/testing/selftests/ublk/test_stress_03.sh
+++ b/tools/testing/selftests/ublk/test_stress_03.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 ublk_io_and_remove()
 {
@@ -10,8 +11,8 @@ ublk_io_and_remove()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -29,24 +30,38 @@ _create_backfile 1 128M
 _create_backfile 2 128M
 
 ublk_io_and_remove 8G -t null -q 4 -z &
+pids+=("$!")
 ublk_io_and_remove 256M -t loop -q 4 -z "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_remove 256M -t stripe -q 4 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 if _have_feature "AUTO_BUF_REG"; then
 	ublk_io_and_remove 8G -t null -q 4 --auto_zc &
+	pids+=("$!")
 	ublk_io_and_remove 256M -t loop -q 4 --auto_zc "${UBLK_BACKFILES[0]}" &
+	pids+=("$!")
 	ublk_io_and_remove 256M -t stripe -q 4 --auto_zc "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+	pids+=("$!")
 	ublk_io_and_remove 8G -t null -q 4 -z --auto_zc --auto_zc_fallback &
-	wait
+	pids+=("$!")
+	_wait_test_pids "${pids[@]}"
+	pids=()
 fi
 
 if _have_feature "PER_IO_DAEMON"; then
 	ublk_io_and_remove 8G -t null -q 4 --auto_zc --nthreads 8 --per_io_tasks &
+	pids+=("$!")
 	ublk_io_and_remove 256M -t loop -q 4 --auto_zc --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[0]}" &
+	pids+=("$!")
 	ublk_io_and_remove 256M -t stripe -q 4 --auto_zc --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+	pids+=("$!")
 	ublk_io_and_remove 8G -t null -q 4 -z --auto_zc --auto_zc_fallback --nthreads 8 --per_io_tasks &
-	wait
+	pids+=("$!")
+	_wait_test_pids "${pids[@]}"
+	pids=()
 fi
 
 _cleanup_test
diff --git a/tools/testing/selftests/ublk/test_stress_04.sh b/tools/testing/selftests/ublk/test_stress_04.sh
index 896eae6..85aaba7 100755
--- a/tools/testing/selftests/ublk/test_stress_04.sh
+++ b/tools/testing/selftests/ublk/test_stress_04.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 ublk_io_and_kill_daemon()
 {
@@ -10,8 +11,8 @@ ublk_io_and_kill_daemon()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -28,24 +29,38 @@ _create_backfile 1 128M
 _create_backfile 2 128M
 
 ublk_io_and_kill_daemon 8G -t null -q 4 -z --no_ublk_fixed_fd &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t loop -q 4 -z --no_ublk_fixed_fd "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t stripe -q 4 -z "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 if _have_feature "AUTO_BUF_REG"; then
 	ublk_io_and_kill_daemon 8G -t null -q 4 --auto_zc &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 256M -t loop -q 4 --auto_zc "${UBLK_BACKFILES[0]}" &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 256M -t stripe -q 4 --auto_zc --no_ublk_fixed_fd "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 8G -t null -q 4 -z --auto_zc --auto_zc_fallback &
-	wait
+	pids+=("$!")
+	_wait_test_pids "${pids[@]}"
+	pids=()
 fi
 
 if _have_feature "PER_IO_DAEMON"; then
 	ublk_io_and_kill_daemon 8G -t null -q 4 --auto_zc --nthreads 8 --per_io_tasks &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 256M -t loop -q 4 --auto_zc --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[0]}" &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 256M -t stripe -q 4 --auto_zc --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+	pids+=("$!")
 	ublk_io_and_kill_daemon 8G -t null -q 4 -z --auto_zc --auto_zc_fallback --nthreads 8 --per_io_tasks &
-	wait
+	pids+=("$!")
+	_wait_test_pids "${pids[@]}"
+	pids=()
 fi
 
 _cleanup_test
diff --git a/tools/testing/selftests/ublk/test_stress_05.sh b/tools/testing/selftests/ublk/test_stress_05.sh
index d6c00c7..baf8e0c 100755
--- a/tools/testing/selftests/ublk/test_stress_05.sh
+++ b/tools/testing/selftests/ublk/test_stress_05.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 if ! _have_program fio; then
 	exit "$UBLK_SKIP_CODE"
@@ -15,8 +16,7 @@ run_io_and_remove()
 	local dev_pid
 	shift 1
 
-	dev_id=$(_add_ublk_dev "$@")
-	_check_add_dev $TID $?
+	dev_id=$(_add_ublk_dev "$@") || return $?
 
 	[ "$UBLK_TEST_QUIET" -eq 0 ] && echo "run ublk IO vs. remove device(ublk add $*)"
 
@@ -40,8 +40,8 @@ ublk_io_and_remove()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 _prep_test "stress" "run IO and remove device with recovery enabled"
@@ -51,33 +51,47 @@ _create_backfile 1 256M
 
 for reissue in $(seq 0 1); do
 	ublk_io_and_remove 8G -t null -q 4 -g -r 1 -i "$reissue" &
+	pids+=("$!")
 	ublk_io_and_remove 256M -t loop -q 4 -g -r 1 -i "$reissue" "${UBLK_BACKFILES[0]}" &
-	wait
+	pids+=("$!")
+	_wait_test_pids "${pids[@]}"
+	pids=()
 done
 
 if _have_feature "ZERO_COPY"; then
 	for reissue in $(seq 0 1); do
 		ublk_io_and_remove 8G -t null -q 4 -z -r 1 -i "$reissue" &
+		pids+=("$!")
 		ublk_io_and_remove 256M -t loop -q 4 -z -r 1 -i "$reissue" "${UBLK_BACKFILES[1]}" &
-		wait
+		pids+=("$!")
+		_wait_test_pids "${pids[@]}"
+		pids=()
 	done
 fi
 
 if _have_feature "AUTO_BUF_REG"; then
 	for reissue in $(seq 0 1); do
 		ublk_io_and_remove 8G -t null -q 4 --auto_zc -r 1 -i "$reissue" &
+		pids+=("$!")
 		ublk_io_and_remove 256M -t loop -q 4 --auto_zc -r 1 -i "$reissue" "${UBLK_BACKFILES[1]}" &
+		pids+=("$!")
 		ublk_io_and_remove 8G -t null -q 4 -z --auto_zc --auto_zc_fallback -r 1 -i "$reissue" &
-		wait
+		pids+=("$!")
+		_wait_test_pids "${pids[@]}"
+		pids=()
 	done
 fi
 
 if _have_feature "PER_IO_DAEMON"; then
 	ublk_io_and_remove 8G -t null -q 4 --nthreads 8 --per_io_tasks -r 1 -i "$reissue" &
+	pids+=("$!")
 	ublk_io_and_remove 256M -t loop -q 4 --nthreads 8 --per_io_tasks -r 1 -i "$reissue" "${UBLK_BACKFILES[0]}" &
+	pids+=("$!")
 	ublk_io_and_remove 8G -t null -q 4 --nthreads 8 --per_io_tasks -r 1 -i "$reissue"  &
+	pids+=("$!")
 fi
-wait
+_wait_test_pids "${pids[@]}"
+pids=()
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_06.sh b/tools/testing/selftests/ublk/test_stress_06.sh
index 9481a27..c6dea85 100755
--- a/tools/testing/selftests/ublk/test_stress_06.sh
+++ b/tools/testing/selftests/ublk/test_stress_06.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 ublk_io_and_remove()
 {
@@ -10,8 +11,8 @@ ublk_io_and_remove()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -25,14 +26,22 @@ _create_backfile 1 128M
 _create_backfile 2 128M
 
 ublk_io_and_remove 8G -t null -q 4 -u &
+pids+=("$!")
 ublk_io_and_remove 256M -t loop -q 4 -u "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_remove 256M -t stripe -q 4 -u "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 ublk_io_and_remove 8G -t null -q 4 -u --nthreads 8 --per_io_tasks &
+pids+=("$!")
 ublk_io_and_remove 256M -t loop -q 4 -u --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_remove 256M -t stripe -q 4 -u --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_07.sh b/tools/testing/selftests/ublk/test_stress_07.sh
index 3e01c03..d42df5d 100755
--- a/tools/testing/selftests/ublk/test_stress_07.sh
+++ b/tools/testing/selftests/ublk/test_stress_07.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 ublk_io_and_kill_daemon()
 {
@@ -10,8 +11,8 @@ ublk_io_and_kill_daemon()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -25,14 +26,22 @@ _create_backfile 1 128M
 _create_backfile 2 128M
 
 ublk_io_and_kill_daemon 8G -t null -q 4 -u --no_ublk_fixed_fd &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t loop -q 4 -u --no_ublk_fixed_fd "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t stripe -q 4 -u "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 ublk_io_and_kill_daemon 8G -t null -q 4 -u --nthreads 8 --per_io_tasks &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t loop -q 4 -u --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t stripe -q 4 -u --nthreads 8 --per_io_tasks "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_08.sh b/tools/testing/selftests/ublk/test_stress_08.sh
index 5f32424..16dde23 100755
--- a/tools/testing/selftests/ublk/test_stress_08.sh
+++ b/tools/testing/selftests/ublk/test_stress_08.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 ublk_io_and_remove()
 {
@@ -10,8 +11,8 @@ ublk_io_and_remove()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -35,10 +36,15 @@ _create_backfile 1 128M
 _create_backfile 2 128M
 
 ublk_io_and_remove 8G -t null -q 4 -b &
+pids+=("$!")
 ublk_io_and_remove 256M -t loop -q 4 --auto_zc -b "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_remove 256M -t stripe -q 4 --auto_zc -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+pids+=("$!")
 ublk_io_and_remove 8G -t null -q 4 -z --auto_zc --auto_zc_fallback -b &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 _cleanup_test
 _show_result $TID $ERR_CODE
diff --git a/tools/testing/selftests/ublk/test_stress_09.sh b/tools/testing/selftests/ublk/test_stress_09.sh
index 64cb8d9..9dd7625 100755
--- a/tools/testing/selftests/ublk/test_stress_09.sh
+++ b/tools/testing/selftests/ublk/test_stress_09.sh
@@ -3,6 +3,7 @@
 
 . "$(cd "$(dirname "$0")" && pwd)"/test_common.sh
 ERR_CODE=0
+pids=()
 
 ublk_io_and_kill_daemon()
 {
@@ -10,8 +11,8 @@ ublk_io_and_kill_daemon()
 	ERR_CODE=$?
 	if [ ${ERR_CODE} -ne 0 ]; then
 		echo "$TID failure: $*"
-		_show_result $TID $ERR_CODE
 	fi
+	return "$ERR_CODE"
 }
 
 if ! _have_program fio; then
@@ -34,10 +35,15 @@ _create_backfile 1 128M
 _create_backfile 2 128M
 
 ublk_io_and_kill_daemon 8G -t null -q 4 -z -b &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t loop -q 4 --auto_zc -b "${UBLK_BACKFILES[0]}" &
+pids+=("$!")
 ublk_io_and_kill_daemon 256M -t stripe -q 4 -b "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
+pids+=("$!")
 ublk_io_and_kill_daemon 8G -t null -q 4 -z --auto_zc --auto_zc_fallback -b &
-wait
+pids+=("$!")
+_wait_test_pids "${pids[@]}"
+pids=()
 
 _cleanup_test
 _show_result $TID $ERR_CODE

[PATCH 3/4] ublk: cancel commits arriving after the quiesce scan

[shao.mingyin]

From: Shao Mingyin <shao.mingyin@zte.com.cn>

QUIESCE_DEV marks the queues canceling and cancels pending commands.
A server can still be processing an I/O when that scan passes its slot.
If it submits COMMIT_AND_FETCH_REQ afterwards, the newly published
command can remain queued without a future request to complete it.
The server waits for that completion and the device remains LIVE.

Share the cancellation claim with the command handler and use it after
processing a commit when the queue is canceling. The cancel lock and
CANCELED flag ensure that the control path and command handler cannot
complete the same command twice.

Fixes: b465ae7b2524 ("ublk: add feature UBLK_F_QUIESCE")
Cc: stable@vger.kernel.org
Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
---
Tested on next-20261105 with CONFIG_BLK_DEV_UBLK=y in a local QEMU/KVM
VM (4 vCPUs, 4 GiB), using real ublk devices and fio. A test-only server
hook held one COMMIT across a successful quiesce, then released it.
In two runs per version, the original kernel left the device LIVE until
a second quiesce; the fixed kernel reached QUIESCED without that retry.
The paired runs used the same userspace probe and no kernel diagnostics.

The four original recovery scripts also passed twice with this change
and a separate batch SQE initialization fix, without C or driver
diagnostic instrumentation or reduced fio load.

 drivers/block/ublk_drv.c |   40 ++++++++++++++++++++++++++--------------
 1 file changed, 26 insertions(+), 14 deletions(-)

diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index 3d322fa1b..082f105f1 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -2792,14 +2792,33 @@ static void ublk_start_cancel(struct ublk_device *ub)
 	ublk_put_disk(disk);
 }
 
+static void ublk_complete_cancelled_cmd(struct ublk_queue *ubq,
+					struct ublk_io *io,
+					unsigned int issue_flags)
+{
+	struct io_uring_cmd *cmd = NULL;
+	bool done;
+
+	spin_lock(&ubq->cancel_lock);
+	done = !!(io->flags & UBLK_IO_FLAG_CANCELED);
+	if (!done) {
+		io->flags |= UBLK_IO_FLAG_CANCELED;
+		/* dependency ordered against smp_wmb() in ublk_prep_cancel() */
+		cmd = READ_ONCE(io->cmd);
+		io->cmd = NULL;
+	}
+	spin_unlock(&ubq->cancel_lock);
+
+	if (!done && cmd)
+		io_uring_cmd_done(cmd, UBLK_IO_RES_ABORT, issue_flags);
+}
+
 static void ublk_cancel_cmd(struct ublk_queue *ubq, u16 tag,
 		unsigned int issue_flags)
 {
 	struct ublk_io *io = &ubq->ios[tag];
 	struct ublk_device *ub = ubq->dev;
-	struct io_uring_cmd *cmd = NULL;
 	struct request *req;
-	bool done;
 
 	if (!(io->flags & UBLK_IO_FLAG_ACTIVE))
 		return;
@@ -2818,18 +2837,7 @@ static void ublk_cancel_cmd(struct ublk_queue *ubq, u16 tag,
 	if (req && blk_mq_request_started(req) && req->tag == tag)
 		return;
 
-	spin_lock(&ubq->cancel_lock);
-	done = !!(io->flags & UBLK_IO_FLAG_CANCELED);
-	if (!done) {
-		io->flags |= UBLK_IO_FLAG_CANCELED;
-		/* dependency ordered against smp_wmb() in ublk_prep_cancel() */
-		cmd = READ_ONCE(io->cmd);
-		io->cmd = NULL;
-	}
-	spin_unlock(&ubq->cancel_lock);
-
-	if (!done && cmd)
-		io_uring_cmd_done(cmd, UBLK_IO_RES_ABORT, issue_flags);
+	ublk_complete_cancelled_cmd(ubq, io, issue_flags);
 }
 
 /*
@@ -3560,6 +3568,10 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
 			req->__sector = addr;
 		if (compl)
 			__ublk_complete_rq(req, io, ublk_dev_need_map_io(ub), NULL);
+
+		/* Quiesce may have scanned this slot before this command arrived. */
+		if (unlikely(READ_ONCE(ubq->canceling)))
+			ublk_complete_cancelled_cmd(ubq, io, issue_flags);
 		break;
 	}
 	case UBLK_IO_NEED_GET_DATA:

[PATCH 4/4] selftests: ublk: clear reused batch SQE command flags

[shao.mingyin]

From: Shao Mingyin <shao.mingyin@zte.com.cn>

Batch FETCH commands set IORING_URING_CMD_MULTISHOT and select a buffer.
When a SQE slot is reused for a PREP or COMMIT command, batch command
initialization overwrites the SQE flags but leaves uring_cmd_flags set.
The multishot flag then disagrees with buffer selection, so
io_uring_cmd_prep() rejects the command with -EINVAL. The completion
assertion aborts kublk, leaving a recovery device QUIESCED or its CLI
waiting for a daemon notification that will never arrive.

Clear rw_flags when initializing a batch command. The FETCH path sets
its multishot flag explicitly afterwards.

Fixes: cb5a6b308700 ("selftests: ublk: handle UBLK_U_IO_FETCH_IO_CMDS")
Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
---
Tested on next-20261105 with CONFIG_BLK_DEV_UBLK=y in a local QEMU/KVM
VM (4 vCPUs, 4 GiB), using real kublk and fio. With the same diagnostic
kernel and batch null workload, the original binary received -EINVAL
for a multishot COMMIT without buffer selection and its daemon exited,
leaving the device QUIESCED. With this change the daemon remained alive
and the device stayed LIVE before deliberate device deletion.

The four recovery scripts also passed twice with this change and a
separate fix for commits arriving after the quiesce cancellation scan.

 tools/testing/selftests/ublk/batch.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/ublk/batch.c b/tools/testing/selftests/ublk/batch.c
index 625a9c32a..fcab2ca9e 100644
--- a/tools/testing/selftests/ublk/batch.c
+++ b/tools/testing/selftests/ublk/batch.c
@@ -241,6 +241,7 @@ static void ublk_init_batch_cmd(struct ublk_thread *t, __u16 q_id,
 
 	sqe->fd	= 0;	/* dev->fds[0] */
 	sqe->opcode	= IORING_OP_URING_CMD;
+	sqe->rw_flags	= 0;
 	sqe->flags	= IOSQE_FIXED_FILE;
 
 	cmd->q_id	= q_id;

Re: [PATCH 2/4] selftests: ublk: propagate stress worker failures

[Ming Lei]

On Wed, Nov 11, 2026 at 10:57:59PM +0800, shao.mingyin@zte.com.cn wrote:
> From: Shao Mingyin <shao.mingyin@zte.com.cn>
> 
> The stress tests update ERR_CODE in background subshells and use a bare
> wait in the parent. A worker can therefore fail while the parent still
> prints PASS and exits successfully. Calling _show_result() from a worker
> also removes backing files shared with other workers.
> 
> Generalize the recovery PID collector and use it to wait for every stress
> worker in the parent. Return errors from the worker helpers, leave result
> reporting and cleanup to the parent, and reset the PID array after each
> batch. Keep failures ahead of skips when collecting worker exit statuses.
> 
> Fixes: bb2cabf23568 ("selftests: ublk: run stress tests in parallel")
> Reported-by: Ming Lei <tom.leiming@gmail.com>
> Link: https://lore.kernel.org/all/avJ0Vn9G7XhZv-XL@fedora-laptop/
> Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
> ---

Reviewed-by: Ming Lei <ming.lei@redhat.com>

Thanks,
Ming

Re: [PATCH 3/4] ublk: cancel commits arriving after the quiesce scan

[Ming Lei]

On Wed, Nov 11, 2026 at 10:58:01PM +0800, shao.mingyin@zte.com.cn wrote:
> From: Shao Mingyin <shao.mingyin@zte.com.cn>
> 
> QUIESCE_DEV marks the queues canceling and cancels pending commands.
> A server can still be processing an I/O when that scan passes its slot.
> If it submits COMMIT_AND_FETCH_REQ afterwards, the newly published
> command can remain queued without a future request to complete it.
> The server waits for that completion and the device remains LIVE.

The issue is real, and it can be reproduced without any test hook:

        kublk add -t fault_inject -q 1 -r 1 --delay_us 3000000
        dd if=/dev/zero of=/dev/ublkbN oflag=direct bs=4k count=1 &
        sleep 1
        kublk quiesce -n N

Without this patch the device stays LIVE after the delayed commit
lands; with it the device reaches QUIESCED. Please add this as a new
recovery selftest in v2, so the fix is covered.

> 
> Share the cancellation claim with the command handler and use it after
> processing a commit when the queue is canceling. The cancel lock and
> CANCELED flag ensure that the control path and command handler cannot
> complete the same command twice.

The commit log needs to cover a few more things:

1) ublk_cancel_cmd() refuses to cancel while the request of this tag is
started, and the new call site skips that check. With zero copy or auto
buffer registration the committed request may still be in flight when
the command is aborted here. I think it is safe, because ->canceling is
already set, so any new request on this tag is aborted in
ublk_queue_rq() and never dispatched to this slot, but please explain
that in the commit log.

2) If ublk_wait_for_idle_io() fails with -EBUSY or -EINTR, ->canceling
is left set and ublk_cancel_dev() isn't called. With this patch the
server gets UBLK_IO_RES_ABORT on its next commit even though QUIESCE_DEV
returned an error. That may be fine, but it is a behavior change and
should be documented.

3) The new check fires whenever ->canceling is set, not only for
QUIESCE_DEV, so the subject and commit log are narrower than the change.

> 
> Fixes: b465ae7b2524 ("ublk: add feature UBLK_F_QUIESCE")
> Cc: stable@vger.kernel.org
> Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
> ---
> Tested on next-20261105 with CONFIG_BLK_DEV_UBLK=y in a local QEMU/KVM
> VM (4 vCPUs, 4 GiB), using real ublk devices and fio. A test-only server
> hook held one COMMIT across a successful quiesce, then released it.
> In two runs per version, the original kernel left the device LIVE until
> a second quiesce; the fixed kernel reached QUIESCED without that retry.
> The paired runs used the same userspace probe and no kernel diagnostics.
> 
> The four original recovery scripts also passed twice with this change
> and a separate batch SQE initialization fix, without C or driver
> diagnostic instrumentation or reduced fio load.
> 
>  drivers/block/ublk_drv.c |   40 ++++++++++++++++++++++++++--------------
>  1 file changed, 26 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
> index 3d322fa1b..082f105f1 100644
> --- a/drivers/block/ublk_drv.c
> +++ b/drivers/block/ublk_drv.c
> @@ -2792,14 +2792,33 @@ static void ublk_start_cancel(struct ublk_device *ub)
>  	ublk_put_disk(disk);
>  }
>  
> +static void ublk_complete_cancelled_cmd(struct ublk_queue *ubq,
> +					struct ublk_io *io,
> +					unsigned int issue_flags)


ublk_drv.c uses "canceled"/"canceling" everywhere, so please name it
ublk_complete_canceled_cmd().


> +{
> +	struct io_uring_cmd *cmd = NULL;
> +	bool done;
> +
> +	spin_lock(&ubq->cancel_lock);
> +	done = !!(io->flags & UBLK_IO_FLAG_CANCELED);
> +	if (!done) {
> +		io->flags |= UBLK_IO_FLAG_CANCELED;
> +		/* dependency ordered against smp_wmb() in ublk_prep_cancel() */
> +		cmd = READ_ONCE(io->cmd);
> +		io->cmd = NULL;
> +	}
> +	spin_unlock(&ubq->cancel_lock);
> +
> +	if (!done && cmd)
> +		io_uring_cmd_done(cmd, UBLK_IO_RES_ABORT, issue_flags);
> +}
> +
>  static void ublk_cancel_cmd(struct ublk_queue *ubq, u16 tag,
>  		unsigned int issue_flags)
>  {
>  	struct ublk_io *io = &ubq->ios[tag];
>  	struct ublk_device *ub = ubq->dev;
> -	struct io_uring_cmd *cmd = NULL;
>  	struct request *req;
> -	bool done;
>  
>  	if (!(io->flags & UBLK_IO_FLAG_ACTIVE))
>  		return;
> @@ -2818,18 +2837,7 @@ static void ublk_cancel_cmd(struct ublk_queue *ubq, u16 tag,
>  	if (req && blk_mq_request_started(req) && req->tag == tag)
>  		return;
>  
> -	spin_lock(&ubq->cancel_lock);
> -	done = !!(io->flags & UBLK_IO_FLAG_CANCELED);
> -	if (!done) {
> -		io->flags |= UBLK_IO_FLAG_CANCELED;
> -		/* dependency ordered against smp_wmb() in ublk_prep_cancel() */
> -		cmd = READ_ONCE(io->cmd);
> -		io->cmd = NULL;
> -	}
> -	spin_unlock(&ubq->cancel_lock);
> -
> -	if (!done && cmd)
> -		io_uring_cmd_done(cmd, UBLK_IO_RES_ABORT, issue_flags);
> +	ublk_complete_cancelled_cmd(ubq, io, issue_flags);
>  }
>  
>  /*
> @@ -3560,6 +3568,10 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
>  			req->__sector = addr;
>  		if (compl)
>  			__ublk_complete_rq(req, io, ublk_dev_need_map_io(ub), NULL);
> +
> +		/* Quiesce may have scanned this slot before this command arrived. */
> +		if (unlikely(READ_ONCE(ubq->canceling)))
> +			ublk_complete_cancelled_cmd(ubq, io, issue_flags);
>  		break;

io->cmd and UBLK_IO_FLAG_ACTIVE are stored here before ->canceling is
read, with no barrier in between, while the cancel scan tests
UBLK_IO_FLAG_ACTIVE locklessly after setting ->canceling. Either side
must see the other's store. That holds in practice because ->canceling
is set under blk_mq_quiesce_queue(), but please say so in the comment
instead of leaving it implicit.

>  	}
>  	case UBLK_IO_NEED_GET_DATA:

UBLK_IO_NEED_GET_DATA publishes io->cmd in the same way. If                                                                                                                                     
ublk_get_data() fails and falls back to requeue, the command is kept
active, and with ->canceling set the requeued request is aborted in
ublk_queue_rq(), so this command looks stranded just like the commit
case. I haven't reproduced that one; can you check, and cover it in v2
if so?

Thanks,
Ming

Re: [PATCH 2/4] selftests: ublk: propagate stress worker failures

[Ming Lei]

On Sat, Nov 14, 2026 at 8:07 PM Ming Lei <tom.leiming@gmail.com> wrote:
>
> On Wed, Nov 11, 2026 at 10:57:59PM +0800, shao.mingyin@zte.com.cn wrote:
> > From: Shao Mingyin <shao.mingyin@zte.com.cn>
> >
> > The stress tests update ERR_CODE in background subshells and use a bare
> > wait in the parent. A worker can therefore fail while the parent still
> > prints PASS and exits successfully. Calling _show_result() from a worker
> > also removes backing files shared with other workers.
> >
> > Generalize the recovery PID collector and use it to wait for every stress
> > worker in the parent. Return errors from the worker helpers, leave result
> > reporting and cleanup to the parent, and reset the PID array after each
> > batch. Keep failures ahead of skips when collecting worker exit statuses.
> >
> > Fixes: bb2cabf23568 ("selftests: ublk: run stress tests in parallel")
> > Reported-by: Ming Lei <tom.leiming@gmail.com>
> > Link: https://lore.kernel.org/all/avJ0Vn9G7XhZv-XL@fedora-laptop/
> > Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
> > ---
>
> Reviewed-by: Ming Lei <ming.lei@redhat.com>

Reviewed-by: Ming Lei <tom.leiming@gmail.com>

Thanks,
Ming Lei

Re: [PATCH 4/4] selftests: ublk: clear reused batch SQE command flags

[Ming Lei]

On Wed, Nov 11, 2026 at 10:58:02PM +0800, shao.mingyin@zte.com.cn wrote:
> From: Shao Mingyin <shao.mingyin@zte.com.cn>
> 
> Batch FETCH commands set IORING_URING_CMD_MULTISHOT and select a buffer.
> When a SQE slot is reused for a PREP or COMMIT command, batch command
> initialization overwrites the SQE flags but leaves uring_cmd_flags set.
> The multishot flag then disagrees with buffer selection, so
> io_uring_cmd_prep() rejects the command with -EINVAL. The completion
> assertion aborts kublk, leaving a recovery device QUIESCED or its CLI
> waiting for a daemon notification that will never arrive.
> 
> Clear rw_flags when initializing a batch command. The FETCH path sets
> its multishot flag explicitly afterwards.
> 
> Fixes: cb5a6b308700 ("selftests: ublk: handle UBLK_U_IO_FETCH_IO_CMDS")
> Signed-off-by: Shao Mingyin <shao.mingyin@zte.com.cn>
> ---
> Tested on next-20261105 with CONFIG_BLK_DEV_UBLK=y in a local QEMU/KVM
> VM (4 vCPUs, 4 GiB), using real kublk and fio. With the same diagnostic
> kernel and batch null workload, the original binary received -EINVAL
> for a multishot COMMIT without buffer selection and its daemon exited,
> leaving the device QUIESCED. With this change the daemon remained alive
> and the device stayed LIVE before deliberate device deletion.
> 
> The four recovery scripts also passed twice with this change and a
> separate fix for commits arriving after the quiesce cancellation scan.
> 
>  tools/testing/selftests/ublk/batch.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tools/testing/selftests/ublk/batch.c b/tools/testing/selftests/ublk/batch.c
> index 625a9c32a..fcab2ca9e 100644
> --- a/tools/testing/selftests/ublk/batch.c
> +++ b/tools/testing/selftests/ublk/batch.c
> @@ -241,6 +241,7 @@ static void ublk_init_batch_cmd(struct ublk_thread *t, __u16 q_id,
>  
>  	sqe->fd	= 0;	/* dev->fds[0] */
>  	sqe->opcode	= IORING_OP_URING_CMD;
> +	sqe->rw_flags	= 0;

Reviewed-by: Ming Lei <tom.leiming@gmail.com>

Thanks,
Ming