* [PATCH blktests 1/5] common/nvme: don't truncate the full log when disconnecting
2026-10-14 7:51 [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Daniel Wagner
@ 2026-10-14 7:51 ` Daniel Wagner
2026-10-14 7:51 ` [PATCH blktests 2/5] nvme/070: test connect with an isolated CPU Daniel Wagner
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Daniel Wagner @ 2026-10-14 7:51 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
From: Daniel Wagner <wagi@kernel.org>
_nvme_disconnect_ctrl and _nvme_disconnect_subsys redirect the output
of nvme disconnect with '&>' into $FULL. This truncates the file and
everything a test logged before the disconnect is lost, e.g. the fio
command lines logged by _run_fio.
Append to $FULL instead.
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
common/nvme | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/common/nvme b/common/nvme
index 8c907c34910d..1b3090e0a5da 100644
--- a/common/nvme
+++ b/common/nvme
@@ -290,7 +290,7 @@ _nvme_disconnect_ctrl() {
if [[ -d "${ctrl_dir}" ]]; then
local ret
- nvme disconnect --device "${ctrl}" &> "${FULL}"
+ nvme disconnect --device "${ctrl}" &>> "${FULL}"
ret=$?
if [[ ${ret} -ne 0 ]]; then
echo "FAIL: nvme disconnect returned error code: ${ret}"
@@ -574,7 +574,7 @@ _nvme_disconnect_subsys() {
esac
done
- nvme disconnect --nqn "${subsysnqn}" &> "${FULL}"
+ nvme disconnect --nqn "${subsysnqn}" &>> "${FULL}"
ret=$?
if [[ ${ret} -ne 0 ]]; then
echo "FAIL: nvme disconnect returned error code: ${ret}"
--
2.56.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH blktests 2/5] nvme/070: test connect with an isolated CPU
2026-10-14 7:51 [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Daniel Wagner
2026-10-14 7:51 ` [PATCH blktests 1/5] common/nvme: don't truncate the full log when disconnecting Daniel Wagner
@ 2026-10-14 7:51 ` Daniel Wagner
2026-10-14 7:51 ` [PATCH blktests 3/5] nvme/071: test I/O queue limits per queue type Daniel Wagner
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Daniel Wagner @ 2026-10-14 7:51 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
From: Daniel Wagner <wagi@kernel.org>
Since kernel commit a46c27026da1 ("blk-mq: don't schedule block kworker
on isolated CPUs") isolated CPUs are removed from the CPU mask of the
hardware contexts. If only isolated CPUs are mapped to a hardware
context, the mask is empty and blk_mq_alloc_request_hctx fails with
-EXDEV. The fabrics transports allocate the Connect command for each
I/O queue this way, so the connect fails, e.g. with isolcpus=domain or
with an isolated cpuset partition.
Isolate the last online CPU with a cpuset partition, connect with the
default number of I/O queues and run verified I/O from a housekeeping
CPU and from the isolated CPU. To make sure the test exercises the
problem, it checks that the hardware context of the isolated CPU has an
empty CPU mask, that the I/O of the isolated CPU runs on that CPU, and
traces nvme_setup_cmd on the queue of the hardware context: the
housekeeping CPUs must not submit commands on it, the isolated CPU must.
The problem is fixed by kernel commit XXX ("blk-mq: use a housekeeping
CPU for passthrough requests on isolated hctxs").
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
tests/nvme/070 | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/070.out | 2 +
2 files changed, 198 insertions(+)
diff --git a/tests/nvme/070 b/tests/nvme/070
new file mode 100755
index 000000000000..6b9d4d80e257
--- /dev/null
+++ b/tests/nvme/070
@@ -0,0 +1,196 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test connecting with an isolated CPU. blk-mq removes the CPUs which are
+# isolated, e.g. with a cpuset isolated partition, from the CPU mask of the
+# hardware contexts. If a hardware context ends up without a CPU, the Connect
+# command for its I/O queue can't be allocated and the connect fails. Run I/O
+# from a housekeeping CPU and from the isolated CPU and trace the queue IDs of
+# the commands to verify that the hardware context of the isolated CPU is used
+# and works.
+#
+# The bug was introduced by commit a46c27026da1 ("blk-mq: don't schedule block
+# kworker on isolated CPUs").
+#
+# Regression test for commit XXX ("blk-mq: use a housekeeping CPU for
+# passthrough requests on isolated hctxs").
+
+. tests/nvme/rc
+. common/cgroup
+
+DESCRIPTION="Test connect with an isolated CPU"
+QUICK=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _require_nvme_trtype_is_fabrics
+ _require_min_cpus 2
+ _have_cgroup2_controller_file cpuset cpuset.cpus.partition
+ _have_tracefs
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+isolate_cpu() {
+ local cpu="$1"
+ local partition
+
+ _init_cgroup2 || return 1
+ _set_attr "+cpuset" "$(_cgroup2_base_dir)/cgroup.subtree_control"
+ _set_attr "${cpu}" "${CGROUP2_DIR}/cpuset.cpus"
+ _set_attr "isolated" "${CGROUP2_DIR}/cpuset.cpus.partition"
+
+ partition=$(cat "${CGROUP2_DIR}/cpuset.cpus.partition")
+ if [[ "${partition}" != "isolated" ]]; then
+ SKIP_REASONS+=("failed to isolate CPU ${cpu}: ${partition}")
+ _exit_cgroup2
+ return 1
+ fi
+
+ return 0
+}
+
+run_io() {
+ local ns="$1"
+
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}"
+}
+
+# Set isolated_hctx to the index of the hardware context the software context
+# of the CPU is mapped to, if the CPU mask of the hardware context is empty.
+find_isolated_hctx() {
+ local nvmedev="$1"
+ local cpu="$2"
+ local ctx
+ local hctx
+
+ for ctx in /sys/class/nvme/"${nvmedev}"/nvme*n*/mq/*/cpu"${cpu}"; do
+ [[ -d "${ctx}" ]] || continue
+ hctx="$(dirname "${ctx}")"
+ if [[ -n "$(cat "${hctx}/cpu_list")" ]]; then
+ SKIP_REASONS+=("hctx of CPU ${cpu} has CPUs: $(cat "${hctx}/cpu_list")")
+ return 1
+ fi
+ isolated_hctx="$(basename "${hctx}")"
+ return 0
+ done
+
+ SKIP_REASONS+=("no hctx found for CPU ${cpu}")
+ return 1
+}
+
+start_trace() {
+ local trace_dir="$1"
+ local ctrl_id="$2"
+ local qid="$3"
+ local event="${trace_dir}/events/nvme/nvme_setup_cmd"
+
+ _set_attr "" "${trace_dir}/trace"
+ _set_attr "ctrl_id == ${ctrl_id} && qid == ${qid}" "${event}/filter"
+ _set_attr 1 "${event}/enable"
+}
+
+# Stop tracing and print the number of traced commands.
+stop_trace() {
+ local trace_dir="$1"
+
+ _set_attr 0 "${trace_dir}/events/nvme/nvme_setup_cmd/enable"
+ grep -c "nvme_setup_cmd:" "${trace_dir}/trace"
+}
+
+check_io() {
+ local nvmedev="$1"
+ local cpu="$2"
+ local trace_dir="$3"
+ local ns
+ local isolated_hctx
+ local qid
+ local nr_cmds
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ find_isolated_hctx "${nvmedev}" "${cpu}" || return 1
+ qid=$((isolated_hctx + 1))
+
+ if [[ ! -d "${trace_dir}/events/nvme/nvme_setup_cmd" ]]; then
+ SKIP_REASONS+=("tracepoint nvme/nvme_setup_cmd does not exist")
+ return 1
+ fi
+
+ # The test runs on the housekeeping CPUs, which must not use the
+ # hardware context of the isolated CPU.
+ start_trace "${trace_dir}" "${nvmedev#nvme}" "${qid}"
+ run_io "${ns}" || echo "FAIL: I/O from housekeeping CPU"
+ nr_cmds=$(stop_trace "${trace_dir}")
+ if [[ "${nr_cmds}" -ne 0 ]]; then
+ echo "FAIL: ${nr_cmds} commands from housekeeping CPUs on qid ${qid}"
+ fi
+
+ # Moving the subshell into the partition confines it to the
+ # isolated CPU, so its I/O uses the hardware context of the
+ # isolated CPU.
+ start_trace "${trace_dir}" "${nvmedev#nvme}" "${qid}"
+ (
+ local allowed
+
+ _set_attr "${BASHPID}" "${CGROUP2_DIR}/cgroup.procs"
+ allowed=$(awk '/^Cpus_allowed_list:/ { print $2 }' \
+ "/proc/${BASHPID}/status")
+ if [[ "${allowed}" != "${cpu}" ]]; then
+ echo "FAIL: subshell runs on CPUs ${allowed}"
+ exit 1
+ fi
+ run_io "${ns}"
+ ) || echo "FAIL: I/O from isolated CPU ${cpu}"
+ nr_cmds=$(stop_trace "${trace_dir}")
+ if [[ "${nr_cmds}" -eq 0 ]]; then
+ echo "FAIL: no commands from isolated CPU ${cpu} on qid ${qid}"
+ fi
+
+ return 0
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cpu
+ local nvmedev
+ local trace_dir="/sys/kernel/debug/tracing/instances/blktests-nvme-070"
+ local skipped=false
+
+ _setup_nvmet
+
+ # Isolate the last online CPU. Without nr_io_queues there is usually
+ # one I/O queue per online CPU, so its hardware context has no other
+ # CPU.
+ cpu=$(awk -F'[-,]' '{ print $NF }' /sys/devices/system/cpu/online)
+ if ! isolate_cpu "${cpu}"; then
+ return 1
+ fi
+
+ _nvmet_target_setup
+
+ if _nvme_connect_subsys; then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+
+ mkdir "${trace_dir}"
+ check_io "${nvmedev}" "${cpu}" "${trace_dir}" || skipped=true
+ rmdir "${trace_dir}"
+
+ _nvme_disconnect_subsys
+ fi
+
+ _nvmet_target_cleanup
+ _exit_cgroup2
+
+ if [[ "${skipped}" = true ]]; then
+ return 1
+ fi
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/070.out b/tests/nvme/070.out
new file mode 100644
index 000000000000..b765a2894652
--- /dev/null
+++ b/tests/nvme/070.out
@@ -0,0 +1,2 @@
+Running nvme/070
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH blktests 3/5] nvme/071: test I/O queue limits per queue type
2026-10-14 7:51 [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Daniel Wagner
2026-10-14 7:51 ` [PATCH blktests 1/5] common/nvme: don't truncate the full log when disconnecting Daniel Wagner
2026-10-14 7:51 ` [PATCH blktests 2/5] nvme/070: test connect with an isolated CPU Daniel Wagner
@ 2026-10-14 7:51 ` Daniel Wagner
2026-10-14 7:51 ` [PATCH blktests 4/5] nvme/072: test poll support without poll queues Daniel Wagner
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Daniel Wagner @ 2026-10-14 7:51 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
From: Daniel Wagner <wagi@kernel.org>
nvmf_nr_io_queues() limits each queue type to the number of online CPUs
when asking the controller for I/O queues, but nvmf_set_io_queues()
used to hand out the granted queues without these limits. With
nr_write_queues larger than the number of online CPUs, the default
queues got more queues than there are CPUs, some hardware contexts had
no CPU mapped and connecting their I/O queues failed with -EXDEV.
Connect with nr_io_queues set to the number of online CPUs and twice as
many write and poll queues, check that every hardware context has a CPU
mapped and that no queue type has more hardware contexts than online
CPUs, and run verified I/O.
The problem is fixed by kernel commit XXX ("nvme-fabrics: limit each
queue type to the number of online CPUs").
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
tests/nvme/071 | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/071.out | 2 ++
2 files changed, 106 insertions(+)
diff --git a/tests/nvme/071 b/tests/nvme/071
new file mode 100755
index 000000000000..2ac5dd761876
--- /dev/null
+++ b/tests/nvme/071
@@ -0,0 +1,104 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test that no queue type gets more I/O queues than there are online CPUs.
+# nvmf_nr_io_queues limits each queue type to the number of online CPUs when
+# asking for I/O queues, but nvmf_set_io_queues used to hand out the granted
+# queues without these limits. With nr_write_queues larger than the number of
+# online CPUs, the default queues got more queues than CPUs, some hardware
+# contexts had no CPU mapped and connecting their I/O queues failed.
+#
+# The bug was introduced for nvme-tcp by commit 6486199378a5 ("nvme-tcp: fix
+# queue mapping when queue count is limited") and for nvme-rdma by commit
+# a249d3066de6 ("nvme-fabrics: add queue setup helpers").
+#
+# Regression test for commit XXX ("nvme-fabrics: limit each queue type to the
+# number of online CPUs").
+
+. tests/nvme/rc
+
+DESCRIPTION="Test I/O queue limits per queue type"
+QUICK=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _have_debugfs
+ _require_nvme_trtype tcp rdma
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+# Check that every hardware context has a software context mapped and that no
+# queue type has more hardware contexts than there are online CPUs.
+check_hctxs() {
+ local nvmedev="$1"
+ local nr_cpus="$2"
+ local disk
+ local hctx
+ local type
+ local -A nr_type=()
+
+ disk=$(basename /sys/class/nvme/"${nvmedev}"/nvme*n*)
+ for hctx in /sys/block/"${disk}"/mq/*; do
+ if ! compgen -G "${hctx}/cpu*[0-9]" > /dev/null; then
+ echo "FAIL: hctx $(basename "${hctx}") has no CPU mapped"
+ fi
+ done
+
+ for hctx in /sys/kernel/debug/block/"${disk}"/hctx*; do
+ type=$(cat "${hctx}/type")
+ nr_type[${type}]=$((${nr_type[${type}]:-0} + 1))
+ done
+ for type in "${!nr_type[@]}"; do
+ if [[ "${nr_type[${type}]}" -gt "${nr_cpus}" ]]; then
+ echo "FAIL: ${nr_type[${type}]} ${type} queues for ${nr_cpus} CPUs"
+ fi
+ done
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cfs_path="${NVMET_CFS}/subsystems/${def_subsysnqn}"
+ local nr_cpus
+ local nvmedev
+ local ns
+
+ _setup_nvmet
+
+ nr_cpus=$(getconf _NPROCESSORS_ONLN)
+
+ _nvmet_target_setup
+
+ # The bug only shows if the target grants a queue per online CPU for
+ # each queue type.
+ if [[ -f "${cfs_path}/attr_qid_max" ]] &&
+ [[ $(cat "${cfs_path}/attr_qid_max") -lt $((3 * nr_cpus)) ]]; then
+ SKIP_REASONS+=("target supports less than $((3 * nr_cpus)) I/O queues")
+ _nvmet_target_cleanup
+ return 1
+ fi
+
+ if _nvme_connect_subsys --nr-io-queues "${nr_cpus}" \
+ --nr-write-queues $((2 * nr_cpus)) \
+ --nr-poll-queues $((2 * nr_cpus)); then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+ check_hctxs "${nvmedev}" "${nr_cpus}"
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}" ||
+ echo "FAIL: fio verify"
+
+ _nvme_disconnect_subsys
+ fi
+
+ _nvmet_target_cleanup
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/071.out b/tests/nvme/071.out
new file mode 100644
index 000000000000..146809b91e0a
--- /dev/null
+++ b/tests/nvme/071.out
@@ -0,0 +1,2 @@
+Running nvme/071
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH blktests 4/5] nvme/072: test poll support without poll queues
2026-10-14 7:51 [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Daniel Wagner
` (2 preceding siblings ...)
2026-10-14 7:51 ` [PATCH blktests 3/5] nvme/071: test I/O queue limits per queue type Daniel Wagner
@ 2026-10-14 7:51 ` Daniel Wagner
2026-10-14 7:51 ` [PATCH blktests 5/5] nvme/073: test reconnect with fewer I/O queues Daniel Wagner
2026-10-20 6:40 ` [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Shin'ichiro Kawasaki
5 siblings, 0 replies; 7+ messages in thread
From: Daniel Wagner @ 2026-10-14 7:51 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
From: Daniel Wagner <wagi@kernel.org>
blk_mq_alloc_tag_set() initializes all queue maps. When nr_poll_queues
is set but the target grants no queues for polling, nvmf_map_queues()
used to leave the poll map in this state, with all CPUs mapped to the
first hardware context. The queue claimed poll support and polled I/O
was sent to a queue which isn't a poll queue.
Limit the target to twice the number of online CPUs, connect with as
many read, write and poll queues as there are online CPUs, so no queues
are left for polling, check that the queue doesn't claim poll support
and run verified I/O.
The problem is fixed by kernel commit XXX ("nvme-fabrics: clear the poll
map if no poll queues are mapped").
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
tests/nvme/072 | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/072.out | 2 ++
2 files changed, 87 insertions(+)
diff --git a/tests/nvme/072 b/tests/nvme/072
new file mode 100755
index 000000000000..72bf80d660f5
--- /dev/null
+++ b/tests/nvme/072
@@ -0,0 +1,85 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test that a controller without poll queues doesn't claim poll support.
+# blk_mq_alloc_tag_set initializes all queue maps. When nr_poll_queues is set
+# but the target grants no queues for polling, nvmf_map_queues used to leave
+# the poll map in this state, so the queue claimed poll support and polled I/O
+# was sent to a queue which isn't a poll queue.
+#
+# The bug was introduced for nvme-tcp by commit 1a9460cef571 ("nvme-tcp:
+# support simple polling") and for nvme-rdma by commit 5651cd3c4336
+# ("nvme-rdma: fix queue mapping when queue count is limited").
+#
+# Regression test for commit XXX ("nvme-fabrics: clear the poll map if no poll
+# queues are mapped").
+
+. tests/nvme/rc
+
+DESCRIPTION="Test poll support without poll queues"
+QUICK=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _require_nvme_trtype tcp rdma
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cfs_path="${NVMET_CFS}/subsystems/${def_subsysnqn}"
+ local nr_cpus
+ local nvmedev
+ local disk
+ local ns
+
+ _setup_nvmet
+
+ nr_cpus=$(getconf _NPROCESSORS_ONLN)
+
+ _nvmet_target_setup
+
+ if [[ ! -f "${cfs_path}/attr_qid_max" ]]; then
+ SKIP_REASONS+=("missing attr_qid_max feature")
+ _nvmet_target_cleanup
+ return 1
+ fi
+ if [[ $(cat "${cfs_path}/attr_qid_max") -lt $((2 * nr_cpus)) ]]; then
+ SKIP_REASONS+=("target supports less than $((2 * nr_cpus)) I/O queues")
+ _nvmet_target_cleanup
+ return 1
+ fi
+
+ # Only grant enough queues for the read and default queues, so no
+ # queues are left for polling.
+ _set_attr $((2 * nr_cpus)) "${cfs_path}/attr_qid_max"
+
+ if _nvme_connect_subsys --nr-io-queues "${nr_cpus}" \
+ --nr-write-queues "${nr_cpus}" \
+ --nr-poll-queues "${nr_cpus}"; then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+ disk=$(basename /sys/class/nvme/"${nvmedev}"/nvme*n*)
+
+ if [[ $(cat "/sys/block/${disk}/queue/io_poll") != 0 ]]; then
+ echo "FAIL: ${disk} claims poll support without poll queues"
+ fi
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}" ||
+ echo "FAIL: fio verify"
+
+ _nvme_disconnect_subsys
+ fi
+
+ _nvmet_target_cleanup
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/072.out b/tests/nvme/072.out
new file mode 100644
index 000000000000..3f765b30a22b
--- /dev/null
+++ b/tests/nvme/072.out
@@ -0,0 +1,2 @@
+Running nvme/072
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH blktests 5/5] nvme/073: test reconnect with fewer I/O queues
2026-10-14 7:51 [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Daniel Wagner
` (3 preceding siblings ...)
2026-10-14 7:51 ` [PATCH blktests 4/5] nvme/072: test poll support without poll queues Daniel Wagner
@ 2026-10-14 7:51 ` Daniel Wagner
2026-10-20 6:40 ` [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Shin'ichiro Kawasaki
5 siblings, 0 replies; 7+ messages in thread
From: Daniel Wagner @ 2026-10-14 7:51 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
From: Daniel Wagner <wagi@kernel.org>
nvmf_set_io_queues() used to only set the io_queues entries for the
queue types it hands out, but the array is reused on every reconnect.
When the target granted fewer queues on a reconnect, the poll queue
count from the previous association survived, nvmf_map_queues() mapped
CPUs to hardware contexts which don't exist anymore and
blk_mq_map_swqueue() crashed with a NULL pointer dereference.
Connect with as many read, write and poll queues as there are online
CPUs, then lower attr_qid_max on the target to twice the number of
online CPUs, which makes the target drop the controller. After the
reconnect with fewer queues, check the queue count, that the queue
doesn't claim poll support and run verified I/O.
The problem is fixed by kernel commits XXX ("nvme-fabrics: reset
io_queues before computing the queue mapping") and XXX ("nvme-fabrics:
clear the poll map if no poll queues are mapped").
Signed-off-by: Daniel Wagner <wagi@kernel.org>
---
tests/nvme/073 | 117 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/073.out | 2 +
2 files changed, 119 insertions(+)
diff --git a/tests/nvme/073 b/tests/nvme/073
new file mode 100755
index 000000000000..1a5afb4fa1df
--- /dev/null
+++ b/tests/nvme/073
@@ -0,0 +1,117 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test reconnecting with fewer I/O queues. nvmf_set_io_queues used to only set
+# the io_queues entries for the queue types it hands out, but the array is
+# reused on every reconnect. When the target grants fewer queues on a
+# reconnect, the poll queue count from the previous association survived,
+# nvmf_map_queues mapped CPUs to hardware contexts which don't exist anymore
+# and blk_mq_map_swqueue crashed. Without the poll map fix, the poll map keeps
+# mapping all CPUs to the first hardware context and the queue claims poll
+# support.
+#
+# The stale io_queues entries were introduced for nvme-tcp by commit
+# 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited") and
+# for nvme-rdma by commit 5651cd3c4336 ("nvme-rdma: fix queue mapping when
+# queue count is limited"). The stale poll map was introduced for nvme-tcp by
+# commit 1a9460cef571 ("nvme-tcp: support simple polling") and for nvme-rdma by
+# commit 5651cd3c4336.
+#
+# Regression test for commits XXX ("nvme-fabrics: reset io_queues before
+# computing the queue mapping") and XXX ("nvme-fabrics: clear the poll map if
+# no poll queues are mapped").
+
+. tests/nvme/rc
+
+DESCRIPTION="Test reconnect with fewer I/O queues"
+QUICK=1
+CHECK_DMESG=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _require_nvme_trtype tcp rdma
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+wait_for_queue_count() {
+ local nvmedev="$1"
+ local queue_count="$2"
+ local file="/sys/class/nvme-fabrics/ctl/${nvmedev}/queue_count"
+ local retries=10
+
+ while [[ $(cat "${file}") -ne $((queue_count + 1)) ]]; do
+ if [[ "${retries}" == 0 ]]; then
+ echo "FAIL: queue count $(cat "${file}") instead of $((queue_count + 1))"
+ return 1
+ fi
+ sleep 1
+ retries=$((retries - 1))
+ done
+
+ return 0
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cfs_path="${NVMET_CFS}/subsystems/${def_subsysnqn}"
+ local nr_cpus
+ local nvmedev
+ local disk
+ local ns
+
+ _setup_nvmet
+
+ nr_cpus=$(getconf _NPROCESSORS_ONLN)
+
+ _nvmet_target_setup
+
+ if [[ ! -f "${cfs_path}/attr_qid_max" ]]; then
+ SKIP_REASONS+=("missing attr_qid_max feature")
+ _nvmet_target_cleanup
+ return 1
+ fi
+ if [[ $(cat "${cfs_path}/attr_qid_max") -lt $((3 * nr_cpus)) ]]; then
+ SKIP_REASONS+=("target supports less than $((3 * nr_cpus)) I/O queues")
+ _nvmet_target_cleanup
+ return 1
+ fi
+
+ if _nvme_connect_subsys --nr-io-queues "${nr_cpus}" \
+ --nr-write-queues "${nr_cpus}" \
+ --nr-poll-queues "${nr_cpus}" \
+ --keep-alive-tmo 1 \
+ --reconnect-delay 2; then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+ disk=$(basename /sys/class/nvme/"${nvmedev}"/nvme*n*)
+ wait_for_queue_count "${nvmedev}" $((3 * nr_cpus))
+
+ # Changing attr_qid_max makes the target drop the controller.
+ # Reconnect with only enough queues for the read and default
+ # queues, so no queues are left for polling.
+ _set_attr $((2 * nr_cpus)) "${cfs_path}/attr_qid_max"
+ wait_for_queue_count "${nvmedev}" $((2 * nr_cpus))
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+
+ if [[ $(cat "/sys/block/${disk}/queue/io_poll") != 0 ]]; then
+ echo "FAIL: ${disk} claims poll support without poll queues"
+ fi
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}" ||
+ echo "FAIL: fio verify"
+
+ _nvme_disconnect_subsys
+ fi
+
+ _nvmet_target_cleanup
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/073.out b/tests/nvme/073.out
new file mode 100644
index 000000000000..ed6690e57868
--- /dev/null
+++ b/tests/nvme/073.out
@@ -0,0 +1,2 @@
+Running nvme/073
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping
2026-10-14 7:51 [PATCH blktests 0/5] nvme: tests for isolated CPUs and fabrics queue mapping Daniel Wagner
` (4 preceding siblings ...)
2026-10-14 7:51 ` [PATCH blktests 5/5] nvme/073: test reconnect with fewer I/O queues Daniel Wagner
@ 2026-10-20 6:40 ` Shin'ichiro Kawasaki
5 siblings, 0 replies; 7+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-10-20 6:40 UTC (permalink / raw)
To: Daniel Wagner; +Cc: linux-block, linux-nvme, Daniel Wagner
On Oct 14, 2026 / 09:51, Daniel Wagner wrote:
> New tests for the bugs fixed by the kernel series "nvme-fabrics: fix
> connects with isolated CPUs and queue mapping bugs":
>
> https://lore.kernel.org/linux-nvme/20261014-nvme-fabrics-isolcpu-v1-0-355156be27b2@kernel.org/
>
> nvme/070 connect with an isolated CPU (cpuset partition), verified
> I/O from a housekeeping and from the isolated CPU, traced on
> the queue of the isolated CPU
> nvme/071 more write and poll queues than online CPUs
> nvme/072 no queues left for polling
> nvme/073 reconnect with fewer I/O queues
>
> nvme/071-073 need write and poll queues and only run for tcp and rdma.
> Without the kernel series all four fail and nvme/073 crashes the kernel,
> with the series they pass. Tested with loop and tcp.
>
> Patch 1 stops _nvme_disconnect_ctrl() and _nvme_disconnect_subsys() from
> truncating $FULL, which dropped everything logged before the disconnect,
> e.g. the fio command lines.
Daniel, thank you for the series. I ran the added test cases, and they work as
expected. With 7.3-rc7 kernel, they failed or hanged. With the kernel fix
series, they all passed.
The first patch in this series is not required because the blktests commit
64cce9a87099 already fixed the problem. Other than that, the patches overall
look good to me. I think left works for this series are as follows:
1) wait for the kernel side fixes applied,
2) modify blktests side commit messages and test script comments to reflect
the kernel side git hashes, and,
3) renumber the test cases
^ permalink raw reply [flat|nested] 7+ messages in thread