[PATCH blktests 0/3] test and handle concurrent IO scheduler switching

[PATCH blktests 0/3] test and handle concurrent IO scheduler switching

[Shin'ichiro Kawasaki]

Until the recent kernel commit 4ff58d6bc9dd ("block: serialize elevator
changes for the same queue using a writer lock") concurrent writes to
the sysfs file queue/scheduler caused KASAN failures [1]. The first
patch of this series adds the regression test case for the fix.

After the kernel commit, writes to the file queue/scheduler can return
-EBUSY. This is rare but can happen when udevd writes to the file at
exactly the same timing as blktests test cases, and it results in
unexpected test failures. The other two patches in this series handle
-EBUSY and avoid the failures.

[1] https://lore.kernel.org/linux-block/20260626004221.711326-1-shinichiro.kawasaki@wdc.com/

Shin'ichiro Kawasaki (3):
  block: add test for concurrent IO scheduler switching
  check: introduce _write_queue_sched() to repeat queue/scheduler writes
  common, block/{015,020,021,027,040}, zbd/012: use _write_queue_sched()

 check                      | 24 ++++++++++++++++++-
 common/dm                  |  2 +-
 common/multipath-over-rdma |  2 +-
 common/rc                  |  5 +---
 tests/block/015            |  2 +-
 tests/block/020            |  2 +-
 tests/block/021            |  2 +-
 tests/block/027            |  2 +-
 tests/block/040            |  3 +--
 tests/block/045            | 49 ++++++++++++++++++++++++++++++++++++++
 tests/block/045.out        |  2 ++
 tests/zbd/012              |  2 +-
 12 files changed, 83 insertions(+), 14 deletions(-)
 create mode 100755 tests/block/045
 create mode 100644 tests/block/045.out

-- 
2.54.0

[PATCH blktests 1/3] block: add test for concurrent IO scheduler switching

[Shin'ichiro Kawasaki]

Add test to confirm that multiple processes can write to queue/scheduler
sysfs attribute of single block device concurrently. This is the fix
confirmation test for the recent bug in the kernel fixed by the kernel
commit 4ff58d6bc9dd ("block: serialize elevator changes for the same
queue using a writer lock").

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 tests/block/045     | 49 +++++++++++++++++++++++++++++++++++++++++++++
 tests/block/045.out |  2 ++
 2 files changed, 51 insertions(+)
 create mode 100755 tests/block/045
 create mode 100644 tests/block/045.out

diff --git a/tests/block/045 b/tests/block/045
new file mode 100755
index 0000000..ef84bc4
--- /dev/null
+++ b/tests/block/045
@@ -0,0 +1,49 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2026 Western Digital Corporation or its affiliates.
+#
+# Switch IO schedulers concurrently to exercise the scheduler change path under
+# contention. This is the regression test for the kernel fix commit 4ff58d6bc9dd
+# ("block: serialize elevator changes for the same queue using a writer lock").
+
+. tests/block/rc
+. common/null_blk
+
+DESCRIPTION="switch schedulers concurrently"
+TIMED=1
+
+switch_schedulers() {
+	local start_time idx
+
+	start_time=$(date +%s)
+	while (( $(date +%s) - start_time <= timeout )); do
+		idx=$((RANDOM % ${#scheds[@]}))
+		{ echo "${scheds[$idx]}" > \
+                       /sys/block/nullb1/queue/scheduler ;} 2> /dev/null
+	done
+}
+
+test() {
+	echo "Running ${TEST_NAME}"
+
+	local timeout pid1 pid2
+	local -a scheds
+
+	if ! _configure_null_blk nullb1 power=1; then
+		return 1
+	fi
+
+	timeout=${TIMEOUT:=30}
+	read -r -a scheds < <(_io_schedulers nullb1)
+
+	switch_schedulers &
+	pid1=$!
+	switch_schedulers &
+	pid2=$!
+
+	wait $pid1 $pid2
+
+	_exit_null_blk
+
+	echo "Test complete"
+}
diff --git a/tests/block/045.out b/tests/block/045.out
new file mode 100644
index 0000000..2301a37
--- /dev/null
+++ b/tests/block/045.out
@@ -0,0 +1,2 @@
+Running block/045
+Test complete
-- 
2.54.0

[PATCH blktests 2/3] check: introduce _write_queue_sched() to repeat queue/scheduler writes

[Shin'ichiro Kawasaki]

After the kernel commit 4ff58d6bc9dd ("block: serialize elevator changes
for the same queue using a writer lock"), concurrent writes to the sysfs
attribute file queue/scheduler of single block device may fail with
-EBUSY. Such failure is observed when udevd writes to the file while
block/005 is running. The write failure causes blktests test case
failures. To avoid it, check the error of the writes, and if the error
is -EBUSY, retry the write.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 check | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/check b/check
index 61524a5..e7eca84 100755
--- a/check
+++ b/check
@@ -385,6 +385,28 @@ _output_test_run() {
 	fi
 }
 
+# Writes to queue/scheduler may fail with -EBUSY when udevd does concurrent
+# write. Retry the writes when -EBUSY is returned.
+_write_queue_sched() {
+	local value="$1"
+	local path="$2"
+	local log
+	local i
+
+	for ((i = 0; i < 5; i++)); do
+		if log=$({ echo "$value" > "$path" ;} 2>&1); then
+			break
+		fi
+		if [[ ! $log =~ "Device or resource busy" ]]; then
+			echo "$log"
+		fi
+	done
+	if ((i > 5)); then
+		echo "echo ${value} > ${path} failed"
+		return 1
+	fi
+}
+
 _register_test_cleanup() {
 	TEST_CLEANUP=$1
 }
@@ -402,7 +424,7 @@ _cleanup() {
 	local key value
 	for key in "${!SYSFS_QUEUE_SAVED[@]}"; do
 		value="${SYSFS_QUEUE_SAVED["$key"]}"
-		echo "$value" >"${key}"
+		_write_queue_sched "$value" "${key}"
 		unset "SYSFS_QUEUE_SAVED[$key]"
 	done
 
-- 
2.54.0

[PATCH blktests 3/3] common, block/{015,020,021,027,040}, zbd/012: use _write_queue_sched()

[Shin'ichiro Kawasaki]

Use _write_queue_sched() to write values to the sysfs attirbute
queue/scheduler to handle -EBUSY.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 common/dm                  | 2 +-
 common/multipath-over-rdma | 2 +-
 common/rc                  | 5 +----
 tests/block/015            | 2 +-
 tests/block/020            | 2 +-
 tests/block/021            | 2 +-
 tests/block/027            | 2 +-
 tests/block/040            | 3 +--
 tests/zbd/012              | 2 +-
 9 files changed, 9 insertions(+), 13 deletions(-)

diff --git a/common/dm b/common/dm
index 14f4265..b0517d6 100644
--- a/common/dm
+++ b/common/dm
@@ -38,7 +38,7 @@ _dm_destination_dev_set_scheduler() {
 			SYSFS_QUEUE_SAVED["$path"]="$(sed \
 					-e 's/.*\[//' -e 's/\].*//' "${path}")"
 		fi
-		echo "${1}" > "${path}"
+		_write_queue_sched "${1}" "${path}"
 	done < <(dmsetup table "$(<"${TEST_DEV_SYSFS}/dm/name")" |
 			 sed -n  's/.* \([0-9]*:[0-9]*\).*/\1/p')
 }
diff --git a/common/multipath-over-rdma b/common/multipath-over-rdma
index 9b72d26..9a4595d 100644
--- a/common/multipath-over-rdma
+++ b/common/multipath-over-rdma
@@ -302,7 +302,7 @@ set_scheduler() {
 			bfq-mq)      s=bfq;;
 		esac
 	fi
-	if ! echo "$s" > "$p"; then
+	if ! _write_queue_sched "$s" "$p"; then
 		echo "Changing scheduler of $b from $(<"$p") into $s failed"
 		return 1
 	fi
diff --git a/common/rc b/common/rc
index de5eb6d..6500533 100644
--- a/common/rc
+++ b/common/rc
@@ -385,10 +385,7 @@ _test_dev_queue_set() {
 	      ${SYSFS_QUEUE_SAVED["$path"]-unset} == unset ]]; then
 		SYSFS_QUEUE_SAVED["$path"]="$(_test_dev_queue_get "$1")"
 	fi
-	if ! echo "$2" >"$path"; then
-		echo "echo $2 > $path failed"
-		return 1
-	fi
+	_write_queue_sched "$2" "$path"
 }
 
 _test_dev_set_scheduler() {
diff --git a/tests/block/015 b/tests/block/015
index afb4b82..31a8135 100755
--- a/tests/block/015
+++ b/tests/block/015
@@ -42,7 +42,7 @@ test() {
 
 	for sched in $(_io_schedulers "$faultb"); do
 		echo "Testing $sched" >> "$FULL"
-		echo "$sched" > /sys/block/"$faultb"/queue/scheduler
+		_write_queue_sched "$sched" /sys/block/"$faultb"/queue/scheduler
 		dd if=/dev/"$faultb" of=/dev/null bs=4K count=$((512 * 1024)) \
 			iflag=direct status=none
 	done
diff --git a/tests/block/020 b/tests/block/020
index 66f380e..22adf46 100755
--- a/tests/block/020
+++ b/tests/block/020
@@ -33,7 +33,7 @@ test() {
 	fi
 	for sched in $(_io_schedulers nullb1); do
 		echo "Testing $sched" >> "$FULL"
-		echo "$sched" > /sys/block/nullb1/queue/scheduler
+		_write_queue_sched "$sched" /sys/block/nullb1/queue/scheduler
 		_fio_perf --bs=4k --ioengine=libaio --iodepth=$iodepth \
 			--numjobs="$(nproc)" --rw=randread --name=async \
 			--filename=/dev/nullb1 --size=1g --direct=1 \
diff --git a/tests/block/021 b/tests/block/021
index 525d707..0d26e76 100755
--- a/tests/block/021
+++ b/tests/block/021
@@ -29,7 +29,7 @@ test() {
 
 	for sched in $(_io_schedulers nullb1); do
 		echo "Testing $sched" >> "$FULL"
-		echo "$sched" > /sys/block/nullb1/queue/scheduler
+		_write_queue_sched "$sched" /sys/block/nullb1/queue/scheduler
 		max_nr="$(cat /sys/block/nullb1/queue/nr_requests)"
 		for ((nr = 4; nr <= max_nr; nr++)); do
 			echo "$nr" > /sys/block/nullb1/queue/nr_requests
diff --git a/tests/block/027 b/tests/block/027
index f59dad2..a208d33 100755
--- a/tests/block/027
+++ b/tests/block/027
@@ -51,7 +51,7 @@ scsi_debug_stress_remove() {
 		# shellcheck disable=SC2207
 		scheds=($(sed 's/[][]//g' "$queue_path/scheduler"))
 		sched_idx=$((cnt % ${#scheds[@]}))
-		echo "${scheds[$sched_idx]}" > "$queue_path/scheduler"
+		_write_queue_sched "${scheds[$sched_idx]}" "$queue_path/scheduler"
 		echo "$cnt" > "$queue_path/../device/queue_depth"
 		((cnt++))
 	done
diff --git a/tests/block/040 b/tests/block/040
index cbf48d4..32e54f0 100755
--- a/tests/block/040
+++ b/tests/block/040
@@ -27,8 +27,7 @@ modify_io_sched() {
 	deadline=$(($(_uptime_s) + TIMEOUT))
 	while [ "$(_uptime_s)" -lt "$deadline" ]; do
 		for sched in $(_io_schedulers "$dev"); do
-			{ echo "$sched" > /sys/block/"$dev"/queue/scheduler ;} \
-				&> /dev/null
+			_write_queue_sched "$sched" /sys/block/"$dev"/queue/scheduler
 			sleep .5
 		done
 	done
diff --git a/tests/zbd/012 b/tests/zbd/012
index c2e47f3..5ec6fd2 100755
--- a/tests/zbd/012
+++ b/tests/zbd/012
@@ -21,7 +21,7 @@ toggle_iosched() {
 
 	while true; do
 		for iosched in none mq-deadline; do
-			echo "${iosched}" > "/sys/class/block/$(basename "$zdev")/queue/scheduler"
+			_write_queue_sched "${iosched}" "/sys/class/block/$(basename "$zdev")/queue/scheduler"
 			sleep .1
 		done
 	done
-- 
2.54.0

Re: [PATCH blktests 0/3] test and handle concurrent IO scheduler switching

[Shin'ichiro Kawasaki]

On Jul 26, 2026 / 14:32, Shin'ichiro Kawasaki wrote:
> Until the recent kernel commit 4ff58d6bc9dd ("block: serialize elevator
> changes for the same queue using a writer lock") concurrent writes to
> the sysfs file queue/scheduler caused KASAN failures [1]. The first
> patch of this series adds the regression test case for the fix.
> 
> After the kernel commit, writes to the file queue/scheduler can return
> -EBUSY. This is rare but can happen when udevd writes to the file at
> exactly the same timing as blktests test cases, and it results in
> unexpected test failures. The other two patches in this series handle
> -EBUSY and avoid the failures.
> 
> [1] https://lore.kernel.org/linux-block/20260626004221.711326-1-shinichiro.kawasaki@wdc.com/
> 
> Shin'ichiro Kawasaki (3):
>   block: add test for concurrent IO scheduler switching
>   check: introduce _write_queue_sched() to repeat queue/scheduler writes
>   common, block/{015,020,021,027,040}, zbd/012: use _write_queue_sched()

FYI, I applied the series. As to the first patch, I renumbered the test case
from block/045 to block/047.