* [PATCH blktests v2 1/6] nvme/076: test I/O queue limits per queue type
2026-11-04 14:25 [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs Daniel Wagner
@ 2026-11-04 14:25 ` Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 2/6] nvme/077: test poll support without poll queues Daniel Wagner
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Daniel Wagner @ 2026-11-04 14:25 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
nvmf_nr_io_queues() limits each queue type to the number of online CPUs
when asking the controller for I/O queues, but nvmf_set_io_queues()
used to hand out the granted queues without these limits. With
nr_write_queues larger than the number of online CPUs, the default
queues got more queues than there are CPUs, some hardware contexts had
no CPU mapped and connecting their I/O queues failed with -EXDEV.
Connect with nr_io_queues set to the number of online CPUs and twice as
many write and poll queues, check that every hardware context has a CPU
mapped and that no queue type has more hardware contexts than online
CPUs, and run verified I/O.
The problem is fixed by kernel commit 99fd0cbae849 ("nvme-fabrics: limit
each queue type to the number of online CPUs").
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
tests/nvme/076 | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/076.out | 2 +
2 files changed, 108 insertions(+)
diff --git a/tests/nvme/076 b/tests/nvme/076
new file mode 100755
index 000000000000..d3389af63bd1
--- /dev/null
+++ b/tests/nvme/076
@@ -0,0 +1,106 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test that no queue type gets more I/O queues than there are online CPUs.
+# nvmf_nr_io_queues limits each queue type to the number of online CPUs when
+# asking for I/O queues, but nvmf_set_io_queues used to hand out the granted
+# queues without these limits. With nr_write_queues larger than the number of
+# online CPUs, the default queues got more queues than CPUs, some hardware
+# contexts had no CPU mapped and connecting their I/O queues failed.
+#
+# The bug was introduced for nvme-tcp by commit 6486199378a5 ("nvme-tcp: fix
+# queue mapping when queue count is limited") and for nvme-rdma by commit
+# a249d3066de6 ("nvme-fabrics: add queue setup helpers").
+#
+# Regression test for commit 99fd0cbae849 ("nvme-fabrics: limit each queue
+# type to the number of online CPUs").
+
+. tests/nvme/rc
+
+DESCRIPTION="Test I/O queue limits per queue type"
+QUICK=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _have_debugfs
+ _require_nvme_trtype tcp rdma
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+# Check that every hardware context has a software context mapped and that no
+# queue type has more hardware contexts than there are online CPUs.
+check_hctxs() {
+ local nvmedev="$1"
+ local nr_cpus="$2"
+ local disk
+ local hctx
+ local type
+ local -A nr_type=()
+
+ disk=$(basename /sys/class/nvme/"${nvmedev}"/nvme*n*)
+ for hctx in /sys/block/"${disk}"/mq/*; do
+ if ! compgen -G "${hctx}/cpu*[0-9]" > /dev/null; then
+ echo "FAIL: hctx $(basename "${hctx}") has no CPU mapped"
+ fi
+ done
+
+ for hctx in /sys/kernel/debug/block/"${disk}"/hctx*; do
+ type=$(cat "${hctx}/type")
+ nr_type[${type}]=$((${nr_type[${type}]:-0} + 1))
+ done
+ for type in "${!nr_type[@]}"; do
+ if [[ "${nr_type[${type}]}" -gt "${nr_cpus}" ]]; then
+ echo "FAIL: ${nr_type[${type}]} ${type} queues for ${nr_cpus} CPUs"
+ fi
+ done
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cfs_path="${NVMET_CFS}/subsystems/${def_subsysnqn}"
+ local nr_cpus
+ local nvmedev
+ local ns
+
+ _setup_nvmet
+
+ nr_cpus=$(getconf _NPROCESSORS_ONLN)
+
+ _nvmet_target_setup
+
+ # The bug only shows if the target grants a queue per online CPU for
+ # each queue type.
+ if [[ -f "${cfs_path}/attr_qid_max" ]] &&
+ [[ $(cat "${cfs_path}/attr_qid_max") -lt $((3 * nr_cpus)) ]]; then
+ SKIP_REASONS+=("target supports less than $((3 * nr_cpus)) I/O queues")
+ _nvmet_target_cleanup
+ return 1
+ fi
+
+ if _nvme_connect_subsys --nr-io-queues "${nr_cpus}" \
+ --nr-write-queues $((2 * nr_cpus)) \
+ --nr-poll-queues $((2 * nr_cpus)); then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+ check_hctxs "${nvmedev}" "${nr_cpus}"
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}" ||
+ echo "FAIL: fio verify"
+
+ _nvme_disconnect_subsys
+ else
+ echo "FAIL: controller not ready"
+ fi
+
+ _nvmet_target_cleanup
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/076.out b/tests/nvme/076.out
new file mode 100644
index 000000000000..f453b79ce2fb
--- /dev/null
+++ b/tests/nvme/076.out
@@ -0,0 +1,2 @@
+Running nvme/076
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH blktests v2 2/6] nvme/077: test poll support without poll queues
2026-11-04 14:25 [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 1/6] nvme/076: test I/O queue limits per queue type Daniel Wagner
@ 2026-11-04 14:25 ` Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 3/6] nvme/078: test reconnect with fewer I/O queues Daniel Wagner
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Daniel Wagner @ 2026-11-04 14:25 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
blk_mq_alloc_tag_set() initializes all queue maps. When nr_poll_queues
is set but the target grants no queues for polling, nvmf_map_queues()
used to leave the poll map in this state, with all CPUs mapped to the
first hardware context. The queue claimed poll support and polled I/O
was sent to a queue which isn't a poll queue.
Limit the target to twice the number of online CPUs, connect with as
many read, write and poll queues as there are online CPUs, so no queues
are left for polling, check that the queue doesn't claim poll support
and run verified I/O.
The problem is fixed by kernel commit 36ede1aa85d5 ("nvme-fabrics: clear
the poll map if no poll queues are mapped").
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
tests/nvme/077 | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/077.out | 2 ++
2 files changed, 89 insertions(+)
diff --git a/tests/nvme/077 b/tests/nvme/077
new file mode 100755
index 000000000000..e57e3416cdde
--- /dev/null
+++ b/tests/nvme/077
@@ -0,0 +1,87 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test that a controller without poll queues doesn't claim poll support.
+# blk_mq_alloc_tag_set initializes all queue maps. When nr_poll_queues is set
+# but the target grants no queues for polling, nvmf_map_queues used to leave
+# the poll map in this state, so the queue claimed poll support and polled I/O
+# was sent to a queue which isn't a poll queue.
+#
+# The bug was introduced for nvme-tcp by commit 1a9460cef571 ("nvme-tcp:
+# support simple polling") and for nvme-rdma by commit 5651cd3c4336
+# ("nvme-rdma: fix queue mapping when queue count is limited").
+#
+# Regression test for commit 36ede1aa85d5 ("nvme-fabrics: clear the poll map
+# if no poll queues are mapped").
+
+. tests/nvme/rc
+
+DESCRIPTION="Test poll support without poll queues"
+QUICK=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _require_nvme_trtype tcp rdma
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cfs_path="${NVMET_CFS}/subsystems/${def_subsysnqn}"
+ local nr_cpus
+ local nvmedev
+ local disk
+ local ns
+
+ _setup_nvmet
+
+ nr_cpus=$(getconf _NPROCESSORS_ONLN)
+
+ _nvmet_target_setup
+
+ if [[ ! -f "${cfs_path}/attr_qid_max" ]]; then
+ SKIP_REASONS+=("missing attr_qid_max feature")
+ _nvmet_target_cleanup
+ return 1
+ fi
+ if [[ $(cat "${cfs_path}/attr_qid_max") -lt $((2 * nr_cpus)) ]]; then
+ SKIP_REASONS+=("target supports less than $((2 * nr_cpus)) I/O queues")
+ _nvmet_target_cleanup
+ return 1
+ fi
+
+ # Only grant enough queues for the read and default queues, so no
+ # queues are left for polling.
+ _set_attr $((2 * nr_cpus)) "${cfs_path}/attr_qid_max"
+
+ if _nvme_connect_subsys --nr-io-queues "${nr_cpus}" \
+ --nr-write-queues "${nr_cpus}" \
+ --nr-poll-queues "${nr_cpus}"; then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+ disk=$(basename /sys/class/nvme/"${nvmedev}"/nvme*n*)
+
+ if [[ $(cat "/sys/block/${disk}/queue/io_poll") != 0 ]]; then
+ echo "FAIL: ${disk} claims poll support without poll queues"
+ fi
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}" ||
+ echo "FAIL: fio verify"
+
+ _nvme_disconnect_subsys
+ else
+ echo "FAIL: controller not ready"
+ fi
+
+ _nvmet_target_cleanup
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/077.out b/tests/nvme/077.out
new file mode 100644
index 000000000000..c7f6ac360a57
--- /dev/null
+++ b/tests/nvme/077.out
@@ -0,0 +1,2 @@
+Running nvme/077
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH blktests v2 3/6] nvme/078: test reconnect with fewer I/O queues
2026-11-04 14:25 [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 1/6] nvme/076: test I/O queue limits per queue type Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 2/6] nvme/077: test poll support without poll queues Daniel Wagner
@ 2026-11-04 14:25 ` Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 4/6] nvme/rc: add helpers to count the commands on a queue Daniel Wagner
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Daniel Wagner @ 2026-11-04 14:25 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
nvmf_set_io_queues() used to only set the io_queues entries for the
queue types it hands out, but the array is reused on every reconnect.
When the target granted fewer queues on a reconnect, the poll queue
count from the previous association survived, nvmf_map_queues() mapped
CPUs to hardware contexts which don't exist anymore and
blk_mq_map_swqueue() crashed with a NULL pointer dereference.
Connect with as many read, write and poll queues as there are online
CPUs, then lower attr_qid_max on the target to twice the number of
online CPUs, which makes the target drop the controller. After the
reconnect with fewer queues, check the queue count, that the queue
doesn't claim poll support and run verified I/O.
The problem is fixed by kernel commits 9588769af5fc ("nvme-fabrics:
reset io_queues before computing the queue mapping") and 36ede1aa85d5
("nvme-fabrics: clear the poll map if no poll queues are mapped").
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
tests/nvme/078 | 119 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/078.out | 2 +
2 files changed, 121 insertions(+)
diff --git a/tests/nvme/078 b/tests/nvme/078
new file mode 100755
index 000000000000..6efeb740e075
--- /dev/null
+++ b/tests/nvme/078
@@ -0,0 +1,119 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test reconnecting with fewer I/O queues. nvmf_set_io_queues used to only set
+# the io_queues entries for the queue types it hands out, but the array is
+# reused on every reconnect. When the target grants fewer queues on a
+# reconnect, the poll queue count from the previous association survived,
+# nvmf_map_queues mapped CPUs to hardware contexts which don't exist anymore
+# and blk_mq_map_swqueue crashed. Without the poll map fix, the poll map keeps
+# mapping all CPUs to the first hardware context and the queue claims poll
+# support.
+#
+# The stale io_queues entries were introduced for nvme-tcp by commit
+# 6486199378a5 ("nvme-tcp: fix queue mapping when queue count is limited") and
+# for nvme-rdma by commit 5651cd3c4336 ("nvme-rdma: fix queue mapping when
+# queue count is limited"). The stale poll map was introduced for nvme-tcp by
+# commit 1a9460cef571 ("nvme-tcp: support simple polling") and for nvme-rdma by
+# commit 5651cd3c4336.
+#
+# Regression test for commits 9588769af5fc ("nvme-fabrics: reset io_queues
+# before computing the queue mapping") and 36ede1aa85d5 ("nvme-fabrics: clear
+# the poll map if no poll queues are mapped").
+
+. tests/nvme/rc
+
+DESCRIPTION="Test reconnect with fewer I/O queues"
+QUICK=1
+CHECK_DMESG=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _require_nvme_trtype tcp rdma
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+wait_for_queue_count() {
+ local nvmedev="$1"
+ local queue_count="$2"
+ local file="/sys/class/nvme-fabrics/ctl/${nvmedev}/queue_count"
+ local retries=10
+
+ while [[ $(cat "${file}") -ne $((queue_count + 1)) ]]; do
+ if [[ "${retries}" == 0 ]]; then
+ echo "FAIL: queue count $(cat "${file}") instead of $((queue_count + 1))"
+ return 1
+ fi
+ sleep 1
+ retries=$((retries - 1))
+ done
+
+ return 0
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cfs_path="${NVMET_CFS}/subsystems/${def_subsysnqn}"
+ local nr_cpus
+ local nvmedev
+ local disk
+ local ns
+
+ _setup_nvmet
+
+ nr_cpus=$(getconf _NPROCESSORS_ONLN)
+
+ _nvmet_target_setup
+
+ if [[ ! -f "${cfs_path}/attr_qid_max" ]]; then
+ SKIP_REASONS+=("missing attr_qid_max feature")
+ _nvmet_target_cleanup
+ return 1
+ fi
+ if [[ $(cat "${cfs_path}/attr_qid_max") -lt $((3 * nr_cpus)) ]]; then
+ SKIP_REASONS+=("target supports less than $((3 * nr_cpus)) I/O queues")
+ _nvmet_target_cleanup
+ return 1
+ fi
+
+ if _nvme_connect_subsys --nr-io-queues "${nr_cpus}" \
+ --nr-write-queues "${nr_cpus}" \
+ --nr-poll-queues "${nr_cpus}" \
+ --keep-alive-tmo 1 \
+ --reconnect-delay 2; then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+ disk=$(basename /sys/class/nvme/"${nvmedev}"/nvme*n*)
+ wait_for_queue_count "${nvmedev}" $((3 * nr_cpus))
+
+ # Changing attr_qid_max makes the target drop the controller.
+ # Reconnect with only enough queues for the read and default
+ # queues, so no queues are left for polling.
+ _set_attr $((2 * nr_cpus)) "${cfs_path}/attr_qid_max"
+ wait_for_queue_count "${nvmedev}" $((2 * nr_cpus))
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+
+ if [[ $(cat "/sys/block/${disk}/queue/io_poll") != 0 ]]; then
+ echo "FAIL: ${disk} claims poll support without poll queues"
+ fi
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}" ||
+ echo "FAIL: fio verify"
+
+ _nvme_disconnect_subsys
+ else
+ echo "FAIL: controller not ready"
+ fi
+
+ _nvmet_target_cleanup
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/078.out b/tests/nvme/078.out
new file mode 100644
index 000000000000..53857c8914c0
--- /dev/null
+++ b/tests/nvme/078.out
@@ -0,0 +1,2 @@
+Running nvme/078
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH blktests v2 4/6] nvme/rc: add helpers to count the commands on a queue
2026-11-04 14:25 [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs Daniel Wagner
` (2 preceding siblings ...)
2026-11-04 14:25 ` [PATCH blktests v2 3/6] nvme/078: test reconnect with fewer I/O queues Daniel Wagner
@ 2026-11-04 14:25 ` Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 5/6] nvme/079: test connect with an isolated CPU Daniel Wagner
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Daniel Wagner @ 2026-11-04 14:25 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
Tests for the queue mapping need to know which I/O queue the commands of
a CPU are sent on. Add helpers which trace the nvme_setup_cmd event for
one queue of a controller and count the commands.
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
tests/nvme/rc | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/tests/nvme/rc b/tests/nvme/rc
index 8abd14eeb112..ea4cd500af9a 100644
--- a/tests/nvme/rc
+++ b/tests/nvme/rc
@@ -640,3 +640,36 @@ _nvme_ctrl_tls_key() {
cat /sys/class/nvme/"$ctrl"/tls_key 2>/dev/null
}
+
+# Check that the trace instance $1 has the nvme_setup_cmd event.
+_nvme_have_setup_cmd_event() {
+ local trace_dir="$1"
+
+ if [[ ! -d "${trace_dir}/events/nvme/nvme_setup_cmd" ]]; then
+ SKIP_REASONS+=("tracepoint nvme/nvme_setup_cmd does not exist")
+ return 1
+ fi
+ return 0
+}
+
+# Start tracing the commands sent on queue $3 of controller $2 in the trace
+# instance $1.
+_nvme_trace_qid_start() {
+ local trace_dir="$1"
+ local ctrl_id="$2"
+ local qid="$3"
+ local event="${trace_dir}/events/nvme/nvme_setup_cmd"
+
+ _set_attr "" "${trace_dir}/trace"
+ _set_attr "ctrl_id == ${ctrl_id} && qid == ${qid}" "${event}/filter"
+ _set_attr 1 "${event}/enable"
+}
+
+# Stop tracing in the trace instance $1 and print the number of traced
+# commands.
+_nvme_trace_qid_stop() {
+ local trace_dir="$1"
+
+ _set_attr 0 "${trace_dir}/events/nvme/nvme_setup_cmd/enable"
+ grep -c "nvme_setup_cmd:" "${trace_dir}/trace"
+}
--
2.56.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH blktests v2 5/6] nvme/079: test connect with an isolated CPU
2026-11-04 14:25 [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs Daniel Wagner
` (3 preceding siblings ...)
2026-11-04 14:25 ` [PATCH blktests v2 4/6] nvme/rc: add helpers to count the commands on a queue Daniel Wagner
@ 2026-11-04 14:25 ` Daniel Wagner
2026-11-04 14:25 ` [PATCH blktests v2 6/6] nvme/080: test connect with offline CPUs Daniel Wagner
2026-11-11 3:48 ` [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or " Shin'ichiro Kawasaki
6 siblings, 0 replies; 9+ messages in thread
From: Daniel Wagner @ 2026-11-04 14:25 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
Since kernel commit a46c27026da1 ("blk-mq: don't schedule block kworker
on isolated CPUs") isolated CPUs are removed from the CPU mask of the
hardware contexts. If only isolated CPUs are mapped to a hardware
context, the mask is empty and blk_mq_alloc_request_hctx fails with
-EXDEV. The fabrics transports allocate the Connect command for each
I/O queue this way, so the connect fails, e.g. with isolcpus=domain or
with an isolated cpuset partition.
Isolate the last online CPU with a cpuset partition, connect with the
default number of I/O queues and run verified I/O from a housekeeping
CPU and from the isolated CPU. To make sure the test exercises the
problem, it checks that the hardware context of the isolated CPU has an
empty CPU mask, that the I/O of the isolated CPU runs on that CPU, and
traces nvme_setup_cmd on the queue of the hardware context: the
housekeeping CPUs must not submit commands on it, the isolated CPU must.
The problem is fixed by kernel commit XXX ("blk-mq: allow passthrough
requests on hctxs without online CPUs").
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
tests/nvme/079 | 176 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/079.out | 2 +
2 files changed, 178 insertions(+)
diff --git a/tests/nvme/079 b/tests/nvme/079
new file mode 100755
index 000000000000..775b864fdd25
--- /dev/null
+++ b/tests/nvme/079
@@ -0,0 +1,176 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test connecting with an isolated CPU. blk-mq removes the CPUs which are
+# isolated, e.g. with a cpuset isolated partition, from the CPU mask of the
+# hardware contexts. If a hardware context ends up without a CPU, the Connect
+# command for its I/O queue can't be allocated and the connect fails. Run I/O
+# from a housekeeping CPU and from the isolated CPU and trace the queue IDs of
+# the commands to verify that the hardware context of the isolated CPU is used
+# and works.
+#
+# The bug was introduced by commit a46c27026da1 ("blk-mq: don't schedule block
+# kworker on isolated CPUs").
+#
+# Regression test for commit XXX ("blk-mq: allow passthrough requests on hctxs
+# without online CPUs").
+
+. tests/nvme/rc
+. common/cgroup
+
+DESCRIPTION="Test connect with an isolated CPU"
+QUICK=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _require_nvme_trtype_is_fabrics
+ _require_min_cpus 2
+ _have_cgroup2_controller_file cpuset cpuset.cpus.partition
+ _have_tracefs
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+isolate_cpu() {
+ local cpu="$1"
+ local partition
+
+ _init_cgroup2 || return 1
+ _set_attr "+cpuset" "$(_cgroup2_base_dir)/cgroup.subtree_control"
+ _set_attr "${cpu}" "${CGROUP2_DIR}/cpuset.cpus"
+ _set_attr "isolated" "${CGROUP2_DIR}/cpuset.cpus.partition"
+
+ partition=$(cat "${CGROUP2_DIR}/cpuset.cpus.partition")
+ if [[ "${partition}" != "isolated" ]]; then
+ SKIP_REASONS+=("failed to isolate CPU ${cpu}: ${partition}")
+ _exit_cgroup2
+ return 1
+ fi
+
+ return 0
+}
+
+run_io() {
+ local ns="$1"
+
+ _run_fio_verify_io --size=32m --filename="/dev/${ns}"
+}
+
+# Set isolated_hctx to the index of the hardware context the software context
+# of the CPU is mapped to, if the CPU mask of the hardware context is empty.
+find_isolated_hctx() {
+ local nvmedev="$1"
+ local cpu="$2"
+ local ctx
+ local hctx
+
+ for ctx in /sys/class/nvme/"${nvmedev}"/nvme*n*/mq/*/cpu"${cpu}"; do
+ [[ -d "${ctx}" ]] || continue
+ hctx="$(dirname "${ctx}")"
+ if [[ -n "$(cat "${hctx}/cpu_list")" ]]; then
+ SKIP_REASONS+=("hctx of CPU ${cpu} has CPUs: $(cat "${hctx}/cpu_list")")
+ return 1
+ fi
+ isolated_hctx="$(basename "${hctx}")"
+ return 0
+ done
+
+ SKIP_REASONS+=("no hctx found for CPU ${cpu}")
+ return 1
+}
+
+check_io() {
+ local nvmedev="$1"
+ local cpu="$2"
+ local trace_dir="$3"
+ local ns
+ local isolated_hctx
+ local qid
+ local nr_cmds
+
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+ find_isolated_hctx "${nvmedev}" "${cpu}" || return 1
+ qid=$((isolated_hctx + 1))
+
+ _nvme_have_setup_cmd_event "${trace_dir}" || return 1
+
+ # The test runs on the housekeeping CPUs, which must not use the
+ # hardware context of the isolated CPU.
+ _nvme_trace_qid_start "${trace_dir}" "${nvmedev#nvme}" "${qid}"
+ run_io "${ns}" || echo "FAIL: I/O from housekeeping CPU"
+ nr_cmds=$(_nvme_trace_qid_stop "${trace_dir}")
+ if [[ "${nr_cmds}" -ne 0 ]]; then
+ echo "FAIL: ${nr_cmds} commands from housekeeping CPUs on qid ${qid}"
+ fi
+
+ # Moving the subshell into the partition confines it to the
+ # isolated CPU, so its I/O uses the hardware context of the
+ # isolated CPU.
+ _nvme_trace_qid_start "${trace_dir}" "${nvmedev#nvme}" "${qid}"
+ (
+ local allowed
+
+ _set_attr "${BASHPID}" "${CGROUP2_DIR}/cgroup.procs"
+ allowed=$(awk '/^Cpus_allowed_list:/ { print $2 }' \
+ "/proc/${BASHPID}/status")
+ if [[ "${allowed}" != "${cpu}" ]]; then
+ echo "FAIL: subshell runs on CPUs ${allowed}"
+ exit 1
+ fi
+ run_io "${ns}"
+ ) || echo "FAIL: I/O from isolated CPU ${cpu}"
+ nr_cmds=$(_nvme_trace_qid_stop "${trace_dir}")
+ if [[ "${nr_cmds}" -eq 0 ]]; then
+ echo "FAIL: no commands from isolated CPU ${cpu} on qid ${qid}"
+ fi
+
+ return 0
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local cpu
+ local nvmedev
+ local trace_dir="/sys/kernel/debug/tracing/instances/blktests-nvme-079"
+ local skipped=false
+
+ _setup_nvmet
+
+ # Isolate the last online CPU. Without nr_io_queues there is usually
+ # one I/O queue per online CPU, so its hardware context has no other
+ # CPU.
+ cpu=$(awk -F'[-,]' '{ print $NF }' /sys/devices/system/cpu/online)
+ if ! isolate_cpu "${cpu}"; then
+ return 1
+ fi
+
+ _nvmet_target_setup
+
+ if _nvme_connect_subsys; then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+
+ mkdir "${trace_dir}"
+ check_io "${nvmedev}" "${cpu}" "${trace_dir}" || skipped=true
+ rmdir "${trace_dir}"
+
+ _nvme_disconnect_subsys
+ else
+ echo "FAIL: controller not ready"
+ fi
+
+ _nvmet_target_cleanup
+ _exit_cgroup2
+
+ if [[ "${skipped}" = true ]]; then
+ return 1
+ fi
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/079.out b/tests/nvme/079.out
new file mode 100644
index 000000000000..59246c004ea3
--- /dev/null
+++ b/tests/nvme/079.out
@@ -0,0 +1,2 @@
+Running nvme/079
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH blktests v2 6/6] nvme/080: test connect with offline CPUs
2026-11-04 14:25 [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs Daniel Wagner
` (4 preceding siblings ...)
2026-11-04 14:25 ` [PATCH blktests v2 5/6] nvme/079: test connect with an isolated CPU Daniel Wagner
@ 2026-11-04 14:25 ` Daniel Wagner
2026-11-11 3:48 ` [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or " Shin'ichiro Kawasaki
6 siblings, 0 replies; 9+ messages in thread
From: Daniel Wagner @ 2026-11-04 14:25 UTC (permalink / raw)
To: Shin'ichiro Kawasaki; +Cc: linux-block, linux-nvme, Daniel Wagner
blk-mq maps all possible CPUs to the hardware contexts, offline CPUs
included. If a hardware context is mapped only to offline CPUs,
blk_mq_alloc_request_hctx has no online CPU to select the software
context from and fails with -EXDEV. The fabrics transports allocate the
Connect command for each I/O queue this way, so connecting fails while
CPUs are offline.
Offline the upper half of the CPUs and connect. Check that a hardware
context is mapped only to offline CPUs, bring them online, run I/O from
one of them and trace that its commands are sent on the queue of that
hardware context.
The bug was introduced by kernel commit 20e4d8139319 ("blk-mq: simplify
queue mapping & schedule with each possisble CPU") and is fixed by
kernel commit XXX ("blk-mq: allow passthrough requests on hctxs without
online CPUs").
Signed-off-by: Daniel Wagner <dwagner@suse.de>
---
tests/nvme/080 | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/nvme/080.out | 2 +
2 files changed, 144 insertions(+)
diff --git a/tests/nvme/080 b/tests/nvme/080
new file mode 100755
index 000000000000..eea4ed237218
--- /dev/null
+++ b/tests/nvme/080
@@ -0,0 +1,142 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (c) 2026 SUSE LLC and contributors
+#
+# Test connecting with offline CPUs. blk-mq maps all possible CPUs to the
+# hardware contexts, the offline ones too. If only offline CPUs are mapped to
+# a hardware context, blk_mq_alloc_request_hctx has no online CPU to take the
+# software context from and fails with -EXDEV. The fabrics transports allocate
+# the Connect command for each I/O queue this way, so the connect fails.
+#
+# Offline the upper half of the CPUs and connect. Check that a hardware context
+# has only offline CPUs mapped, bring them online, run I/O from one of them and
+# trace the queue IDs of the commands to verify that the hardware context is
+# used and works.
+#
+# The bug was introduced by commit 20e4d8139319 ("blk-mq: simplify queue
+# mapping & schedule with each possisble CPU").
+#
+# Regression test for commit XXX ("blk-mq: allow passthrough requests on hctxs
+# without online CPUs").
+
+. tests/nvme/rc
+. common/cpuhotplug
+
+DESCRIPTION="Test connect with offline CPUs"
+QUICK=1
+
+requires() {
+ _nvme_requires
+ _have_fio
+ _have_loop
+ _require_nvme_trtype_is_fabrics
+ _require_min_cpus 4
+ _have_cpu_hotplug
+ _have_tracefs
+}
+
+set_conditions() {
+ _set_nvme_trtype "$@"
+}
+
+offline_upper_cpus() {
+ local -a cpus
+ local cpu
+
+ mapfile -t cpus < <(printf '%s\n' "${HOTPLUGGABLE_CPUS[@]}" | sort -n)
+ for cpu in "${cpus[@]:$((${#cpus[@]} / 2))}"; do
+ _offline_cpu "${cpu}"
+ done
+}
+
+# Print the hctx directory and the CPUs of the first hardware context which
+# only has offline CPUs mapped.
+find_offline_hctx() {
+ local disk="$1"
+ local hctx
+ local cpu
+ local cpus
+
+ for hctx in /sys/block/"${disk}"/mq/*; do
+ cpus=$(tr -d ' ' < "${hctx}/cpu_list" | tr ',' ' ')
+ for cpu in ${cpus}; do
+ if [[ $(cat "/sys/devices/system/cpu/cpu${cpu}/online" \
+ 2>/dev/null || echo 1) == 1 ]]; then
+ continue 2
+ fi
+ done
+ echo "$(basename "${hctx}") ${cpus}"
+ return 0
+ done
+ return 1
+}
+
+test() {
+ echo "Running ${TEST_NAME}"
+
+ local nvmedev
+ local disk
+ local ns
+ local hctx
+ local cpu_list
+ local -a cpus
+ local cpu
+ local nr_cmds
+ local trace_dir="/sys/kernel/debug/tracing/instances/blktests-nvme-080"
+ local skipped=false
+
+ _setup_nvmet
+
+ offline_upper_cpus
+
+ _nvmet_target_setup
+
+ if _nvme_connect_subsys; then
+ _nvmf_wait_for_state "${def_subsysnqn}" "live" || echo FAIL
+ nvmedev=$(_find_nvme_dev "${def_subsysnqn}")
+ disk=$(basename /sys/class/nvme/"${nvmedev}"/nvme*n*)
+ ns=$(_find_nvme_ns "${def_subsys_uuid}")
+
+ # The CPUs are spread over the hardware contexts in topology
+ # order, so usually the upper ones end up without online CPU.
+ if read -r hctx cpu_list < <(find_offline_hctx "${disk}"); then
+ read -r -a cpus <<< "${cpu_list}"
+ for cpu in "${HOTPLUGGABLE_CPUS[@]}"; do
+ _online_cpu "${cpu}"
+ done
+
+ cpu=${cpus[0]}
+ mkdir "${trace_dir}"
+ if _nvme_have_setup_cmd_event "${trace_dir}"; then
+ _nvme_trace_qid_start "${trace_dir}" \
+ "${nvmedev#nvme}" $((hctx + 1))
+ _run_fio_verify_io --size=32m \
+ --filename="/dev/${ns}" \
+ --cpus_allowed="${cpu}" ||
+ echo "FAIL: fio verify on CPU ${cpu}"
+ nr_cmds=$(_nvme_trace_qid_stop "${trace_dir}")
+ if [[ "${nr_cmds}" -eq 0 ]]; then
+ echo "FAIL: no commands from CPU ${cpu} on hctx ${hctx}"
+ fi
+ else
+ skipped=true
+ fi
+ rmdir "${trace_dir}"
+ else
+ SKIP_REASONS+=("no hardware context with only offline CPUs")
+ skipped=true
+ fi
+
+ _nvme_disconnect_subsys
+ else
+ echo "FAIL: controller not ready"
+ fi
+
+ _nvmet_target_cleanup
+
+ if [[ "${skipped}" = true ]]; then
+ return 1
+ fi
+
+ echo "Test complete"
+}
diff --git a/tests/nvme/080.out b/tests/nvme/080.out
new file mode 100644
index 000000000000..15119b8d7a83
--- /dev/null
+++ b/tests/nvme/080.out
@@ -0,0 +1,2 @@
+Running nvme/080
+Test complete
--
2.56.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs
2026-11-04 14:25 [PATCH blktests v2 0/6] nvme: tests for fabrics queue mapping and isolated or offline CPUs Daniel Wagner
` (5 preceding siblings ...)
2026-11-04 14:25 ` [PATCH blktests v2 6/6] nvme/080: test connect with offline CPUs Daniel Wagner
@ 2026-11-11 3:48 ` Shin'ichiro Kawasaki
2026-11-12 9:07 ` Daniel Wagner
6 siblings, 1 reply; 9+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-11-11 3:48 UTC (permalink / raw)
To: Daniel Wagner; +Cc: linux-block, linux-nvme
On Nov 04, 2026 / 15:25, Daniel Wagner wrote:
> Connecting an NVMe over Fabrics controller fails or crashes the kernel
> in several cases in current upstream. These tests reproduce each case
> and verify the fix.
>
> Queue mapping bugs, fixed in mainline:
>
> nvme/076 More write and poll queues than online CPUs requested. A
> queue type got more hardware contexts than online CPUs, a
> hardware context without CPU failed its Connect command with
> -EXDEV. Fixed by 99fd0cbae849 ("nvme-fabrics: limit each
> queue type to the number of online CPUs").
> nvme/077 No queues left for polling. The poll map kept the default
> mapping, the queue claimed poll support and polled I/O went
> to a queue which isn't a poll queue. Fixed by 36ede1aa85d5
> ("nvme-fabrics: clear the poll map if no poll queues are
> mapped").
> nvme/078 Reconnect with fewer I/O queues. Stale io_queues entries
> mapped the poll queues beyond the hardware contexts and
> blk_mq_map_swqueue crashed. Fixed by 9588769af5fc
> ("nvme-fabrics: reset io_queues before computing the queue
> mapping") and 36ede1aa85d5.
Daniel, thanks for the v2 patch. I applied the first three patches for
the test cases above. Please note that I renumbered them to nvme/078-080.
The other three patches look good. IIUC, they depend on another kernel patch
[*]. I would like to wait for it applied, then take another look on them.
[*] https://lore.kernel.org/linux-block/20261104-nvme-fabrics-isolcpu-v3-1-41953b5dfba0@suse.de/
^ permalink raw reply [flat|nested] 9+ messages in thread