[PATCH v2 0/2] Test multipath and marginal ports

[PATCH v2 0/2] Test multipath and marginal ports

[Jesse Taube]

Tests for the upcoming nvme-fc: FPIN link integrity handling set.
It tests for various multipath and marginal port
scenarios, while confirming the port usage and state. The test is
intended to emulate receiving an FPIN event in a multipath environment.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=220329
Link: https://github.com/linux-blktests/blktests/pull/264

Jesse Taube (2):
  nvme: Add _setup_nvmet_port_marginal
  nvme/070: Test multipath and marginal ports

 common/nvme        |  31 +++
 tests/nvme/070     | 613 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/070.out |  43 ++++
 3 files changed, 687 insertions(+)
 create mode 100755 tests/nvme/070
 create mode 100644 tests/nvme/070.out

-- 
2.54.0

[PATCH v2 1/2] nvme: Add _setup_nvmet_port_marginal

[Jesse Taube]

Adds functionality to set up a NVME port in marginal state.
This is useful for testing the behavior of the NVME driver when a port
is in a marginal state.

Signed-off-by: Jesse Taube <jtaubepe@redhat.com>
Reviewed-by: John Meneghini <jmeneghi@redhat.com>
Tested-by: John Meneghini <jmeneghi@redhat.com>
---
V1 -> V2:
 - Added john's review and test-by tags
 - Use != instead of ! ==
 - Wrap lines over 80col
---
 common/nvme | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/common/nvme b/common/nvme
index f399937..a6d6b28 100644
--- a/common/nvme
+++ b/common/nvme
@@ -153,6 +153,16 @@ _nvme_fcloop_add_tport() {
 	echo "wwnn=${wwnn},wwpn=${wwpn}" > ${loopctl}/add_target_port
 }
 
+_nvme_fcloop_set_rport_marginal() {
+	local wwnn="$1"
+	local wwpn="$2"
+	local marginal="$3"
+	local loopctl=/sys/class/fcloop/ctl
+
+	echo "wwnn=${wwnn},wwpn=${wwpn},marginal=${marginal}" \
+	     > ${loopctl}/set_marginal_rport
+}
+
 _nvme_fcloop_del_rport() {
 	local local_wwnn="$1"
 	local local_wwpn="$2"
@@ -763,6 +773,27 @@ _create_nvmet_port() {
 	echo "${port}"
 }
 
+_setup_nvmet_port_marginal() {
+	local -i port="$1"
+	local state="${2}"
+
+	if [[ "${nvme_trtype}" != "fc" ]]; then
+		echo "FAIL _setup_nvmet_port_marginal() only supports fc transport"
+		exit 1
+	fi
+
+	if [[ "${state}" == "live" ]]; then
+		_nvme_fcloop_set_rport_marginal "$(_remote_wwnn $port)" \
+					"$(_remote_wwpn $port)" 0
+	elif [[ "${state}" == "marginal" ]]; then
+		_nvme_fcloop_set_rport_marginal "$(_remote_wwnn $port)" \
+					"$(_remote_wwpn $port)" 1
+	else
+		echo "FAIL _setup_nvmet_port_marginal() invalid state: ${state}"
+		exit 1
+	fi
+}
+
 _setup_nvmet_port_ana() {
 	local port="$1"
 	local anagrpid="${2:-$def_anagrpid}"
-- 
2.54.0

[PATCH v2 2/2] nvme/070: Test multipath and marginal ports

[Jesse Taube]

Add tests/nvme/070 to test various multipath and marginal port
scenarios, while confirming the port useage and state. This test is
intended to emulate receiving an FPIN event in a multipath environment.

Co-authored-by: John Menehgini <jmeneghi@redhat.com>
Signed-off-by: Jesse Taube <jtaubepe@redhat.com>
Signed-off-by: John Meneghini <jmeneghi@redhat.com>
Tested-by: John Meneghini <jmeneghi@redhat.com>
---
V1 -> V2:
 - Add John's tags
 - Add _rport_check_use_opt
 - Add _rport_is_not_in_use to allow for checking with a timeout
 - Add _rport_num_in_use - Wrap lines over 80col
 - Add timeout functionality to _rport_is_marginal, is_online,
     in_use.
 - Fix path state setup to have one optimized and one non-optimized port
     for each host
 - Fix test_set_all_marginal to correctly check non-numa iopolicy
 - Remove _rport_check_opt as it was redundant
 - Rename _rport_optimized to _rport_is_optimized
 - Skip test if /sys/class/fcloop/ctl/set_marginal_rport does not exist
 - Use grep --quiet instead of grep -q
 - Use local -a for arrays
---
 tests/nvme/070     | 613 +++++++++++++++++++++++++++++++++++++++++++++
 tests/nvme/070.out |  43 ++++
 2 files changed, 656 insertions(+)
 create mode 100755 tests/nvme/070
 create mode 100644 tests/nvme/070.out

diff --git a/tests/nvme/070 b/tests/nvme/070
new file mode 100755
index 0000000..f8117ef
--- /dev/null
+++ b/tests/nvme/070
@@ -0,0 +1,613 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-3.0+
+# Copyright (C) 2024 John Meneghini <jmeneghi@redhat.com>
+#
+# Test nvme-fc marginal path handling with fcloop
+
+. tests/nvme/rc
+
+TIMEOUT=10
+
+DESCRIPTION="test nvme-fc marginal path handling"
+
+requires() {
+	_nvme_requires
+	_have_loop
+	_have_fio
+	_require_nvme_trtype fc
+}
+
+set_conditions() {
+	_set_nvme_trtype "$@"
+}
+
+_subsys_rport_addr() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	cat "$RPORT"/address
+	# traddr=nn-0x2047d039ea98949e:pn-0x2048d039ea98949e,host_traddr=nn-0x200000109b9b7e4e:pn-0x100000109b9b7e4e
+}
+
+_subsys_get_port() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local port
+	local address
+
+	address=$(_subsys_rport_addr "$RPORT")
+	port=$(echo "$address" | sed -n 's/.*pn-\(.*\),.*/\1/p')
+	echo $(( port - $(_remote_wwpn 0) ))
+	# a number representing the port number, e.g. 0, 1, 2, 3
+}
+
+_rport_set_iopolicy() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local POLICY=$2 # numa queue-depth round-robin
+
+	echo "$POLICY" | sudo tee "$RPORT"/iopolicy > /dev/null
+	# Returns none
+}
+
+_rport_set_marginal() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	_setup_nvmet_port_marginal "$(_subsys_get_port "$RPORT")" "marginal"
+}
+
+_rport_set_online() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	_setup_nvmet_port_marginal "$(_subsys_get_port "$RPORT")" "live"
+}
+
+_rport_is_online_raw() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	[[ "$(cat "$RPORT"/state)" == "live" ]]
+	# Returns Success or Fail
+}
+
+_rport_is_marginal_raw() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	[[ "$(cat "$RPORT"/state)" == "marginal" ]]
+	# Returns Success or Fail
+}
+
+_rport_in_use_raw() {
+	local SUBSYS_PATH=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# there should only be one "nvme3c9n1" afaik
+	[[ "$(cat "$SUBSYS_PATH"/nvme*/stat | awk '{print $9}')" != "0" ]]
+	# Returns exit code 0 if in use, 1 if not in use
+}
+
+_rport_is_optimized() {
+	local SUBSYS_PATH=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# there should only be one "nvme3c9n1" afaik
+	[[ "$(cat "$SUBSYS_PATH"/nvme*/ana_state)" == "optimized" ]]
+	# Returns exit code 0 if optimized, 1 if not optimized
+}
+
+_rport_is_online() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -i time_passed=0
+
+	for (( time_passed=0; time_passed < TIMEOUT; time_passed++ )); do
+		_rport_is_online_raw "$RPORT" && return 0 || true
+		sleep 1
+	done
+	return 1
+	# Returns Success or Fail
+}
+
+_rport_is_marginal() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -i time_passed=0
+
+	for (( time_passed=0; time_passed < TIMEOUT; time_passed++ )); do
+		_rport_is_marginal_raw "$RPORT" && return 0 || true
+		sleep 1
+	done
+	return 1
+	# Returns Success or Fail
+}
+
+_rport_in_use() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -i time_passed=0
+
+	for (( time_passed=0; time_passed < TIMEOUT; time_passed++ )); do
+		_rport_in_use_raw "$RPORT" && return 0 || true
+		sleep 1
+	done
+	return 1
+	# Returns exit code 0 if in use, 1 if not in use
+}
+
+_rport_not_in_use() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -i time_passed=0
+
+	for (( time_passed=0; time_passed < TIMEOUT; time_passed++ )); do
+		_rport_in_use_raw "$RPORT" || return 0
+		sleep 1
+	done
+	return 1
+	# Returns exit code 0 if in not use, 1 if in use
+}
+
+# check if link is marginal or not
+_rport_check_online() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local STATE=$2 # "Online"
+
+	if [[ "$STATE" == "Online" ]]; then
+		if ! _rport_is_online "$RPORT"; then
+			echo  FC port \("$RPORT"\) is not online, expteced online.
+			return 1
+		fi
+	else
+		if ! _rport_is_marginal "$RPORT"; then
+			echo FC port \("$RPORT"\) is not marginal, expteced marginal.
+			return 1
+		fi
+	fi
+	# Returns Success or Fail
+}
+
+# check if link is in use
+_rport_check_use() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local STATE=$2 # "Online"
+
+	if [[ "$STATE" == "Online" ]]; then
+		if ! _rport_in_use "$RPORT" ; then
+			echo FC port on \("$RPORT"\) is not being used, expected use.
+			return 1
+		fi
+	else
+		if ! _rport_not_in_use "$RPORT" ; then
+			echo FC port on \("$RPORT"\) is being used, expected no use.
+			return 1
+		fi
+	fi
+	# Returns Success or Fail
+}
+
+# check if all link is in state and use state if optimized
+_rport_check_use_opt() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# Only optimized paths will be in use
+	if _rport_is_optimized "$RPORT"; then
+		_rport_check_use "$RPORT" Online || return 1
+	else
+		_rport_check_use "$RPORT" Marginal || return 1
+	fi
+	# Returns Success or Fail
+}
+
+# check if all link is in state and use state
+_rport_check() {
+	local RPORT=$1 # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local STATE=$2 # "Online"
+
+	_rport_check_online "$RPORT" "$STATE" || return 1
+	_rport_check_use "$RPORT" "$STATE"
+	# Returns Success or Fail
+}
+
+_rport_num_in_use() {
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -i num_in_use=0
+
+	# Check if one of the marginal paths is in use
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		_rport_in_use_raw "$subsys_path" || continue
+		num_in_use=$((num_in_use + 1))
+	done
+
+	echo "$num_in_use"
+	# Returns number of links in use
+}
+
+# Check if one of the paths is in use
+_rport_check_one_use() {
+	local STATE=$1
+	shift
+	local -a RPORTS_PATHS=("$@") #  /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -i time_passed=0
+	local -i count_in_use=0
+
+	for (( time_passed=0; time_passed < TIMEOUT; time_passed++ )); do
+		# Check if one of the online paths is in use
+		count_in_use=$(_rport_num_in_use "${RPORTS_PATHS[@]}")
+		if [ $count_in_use -eq 1 ]; then
+			return 0
+		fi
+		sleep 1
+	done
+
+	if [ $count_in_use -gt 1 ]; then
+		echo More than one FC port is being used, expected only one in use when all are "$STATE" in numa mode.
+		return 1
+	fi
+
+	echo No FC ports are being used, expected atleast one in use when all are "$STATE" in numa mode.
+	# None of the online paths were in use
+	return 1
+
+	# Returns Success or Fail
+}
+
+test_set_all_online() {
+	local IOPOLICY=$1 # "numa" "queue-depth" "round-robin"
+	shift
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		_rport_set_online "$subsys_path" || return 1
+		_rport_check_online "$subsys_path" Online || return 1
+	done
+
+	if [ "$IOPOLICY" == "numa" ]; then
+		_rport_check_one_use online "${RPORTS_PATHS[@]}" || return 1
+	else
+		for subsys_path in "${RPORTS_PATHS[@]}"; do
+			# Only optimized paths will be in use
+			_rport_check_use_opt "$subsys_path" || return 1
+		done
+	fi
+}
+
+test_set_one_host_marginal() {
+	local HOST=$1 # host1
+	shift
+	local -a ARGS=("$@")
+	local RPORTS_CNT="$(( $# / 2 ))"
+	local PATHS_POS # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local HOSTS_POS # host1 host2
+
+	for (( PATHS_POS=0; PATHS_POS < RPORTS_CNT; PATHS_POS++ )); do
+		HOSTS_POS="$(( RPORTS_CNT + PATHS_POS ))"
+		if [[ "${ARGS[$HOSTS_POS]}" == "$HOST" ]]; then
+			_rport_set_marginal "${ARGS[$PATHS_POS]}" || return 1
+		fi
+	done
+
+	# check if all links are of expected state on hosts
+	for (( PATHS_POS=0; PATHS_POS < RPORTS_CNT; PATHS_POS++ )); do
+		HOSTS_POS="$(( RPORTS_CNT + PATHS_POS ))"
+		if [[ "${ARGS[$HOSTS_POS]}" == "$HOST" ]]; then
+			_rport_check "${ARGS[$PATHS_POS]}" "Marginal" || return 1
+		else
+			_rport_check_online "${ARGS[$PATHS_POS]}" "Online" || return 1
+			_rport_check_use_opt "${ARGS[$PATHS_POS]}" || return 1
+		fi
+	done
+}
+
+test_set_all_marginal() {
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		_rport_set_marginal "$subsys_path" || return 1
+		_rport_check_online "$subsys_path" Marginal || return 1
+	done
+
+	if [ "$IOPOLICY" == "numa" ]; then
+		_rport_check_one_use marginal "${RPORTS_PATHS[@]}" || return 1
+	else
+		for subsys_path in "${RPORTS_PATHS[@]}"; do
+			# Only optimized paths will be in use
+			_rport_check_use_opt "$subsys_path" Marginal || return 1
+		done
+	fi
+}
+
+test_set_one_non_optimized_online() {
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# Set one non-optimized online
+	# First set one non-optimized online to online then check optimized paths
+	local next_marginal=0
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		if ! _rport_is_optimized "$subsys_path"; then
+			if [ $next_marginal == 1 ]; then
+				_rport_check "$subsys_path" Marginal || return 1
+				break
+			fi
+			_rport_set_online "$subsys_path" || return 1
+			_rport_check "$subsys_path" Online || return 1
+			next_marginal=1
+		fi
+	done
+	# Check optimized paths
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		if _rport_is_optimized "$subsys_path"; then
+			# Optimized paths will not be in use
+			_rport_check "$subsys_path" Marginal || return 1
+		fi
+	done
+}
+
+test_set_all_non_optimized_online() {
+	local IOPOLICY=$1 # "numa" "queue-depth" "round-robin"
+	shift
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# Set all non-optimized online
+	local numa_in_use=0
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		if _rport_is_optimized "$subsys_path"; then
+			_rport_check "$subsys_path" Marginal || return 1
+		else
+			_rport_set_online "$subsys_path" || return 1
+			if [ "$IOPOLICY" == "numa" ] && [ $numa_in_use == 0 ]; then
+				_rport_check "$subsys_path" Online || return 1
+				numa_in_use=1
+			elif [ "$IOPOLICY" == "numa" ] && [ $numa_in_use == 1 ]; then
+				_rport_check_online "$subsys_path" Online || return 1
+				# Only one of the online paths should be in use
+				_rport_check_use "$subsys_path" Marginal || return 1
+			else
+				_rport_check "$subsys_path" Online || return 1
+			fi
+		fi
+	done
+}
+
+set_one_optimized_online() {
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# Set one optimized online
+	local next_marginal=0
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		if _rport_is_optimized "$subsys_path"; then
+			if [ $next_marginal == 1 ]; then
+				_rport_check "$subsys_path" Marginal || return 1
+				break
+			fi
+			_rport_set_online "$subsys_path" || return 1
+			_rport_check "$subsys_path" Online || return 1
+			next_marginal=1
+		fi
+	done
+}
+
+test_set_all_non_one_optimized_online() {
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# Set one optimized online
+	# First set one optimized online to online then check non-optimized paths
+	set_one_optimized_online "${RPORTS_PATHS[@]}" || return 1
+
+	# Check non-optimized paths
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		if ! _rport_is_optimized "$subsys_path"; then
+			_rport_check_online "$subsys_path" Online || return 1
+			# Only optimized paths will be in use
+			_rport_check_use "$subsys_path" Marginal || return 1
+		fi
+	done
+}
+
+test_set_one_optimized_online() {
+	local -a RPORTS_PATHS=("$@") # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+
+	# Set one optimized online
+	# First set one optimized online to online then check non-optimized paths
+	set_one_optimized_online "${RPORTS_PATHS[@]}" || return 1
+
+	# Check non-optimized paths
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		if ! _rport_is_optimized "$subsys_path"; then
+			# Only optimized paths will be in use
+			_rport_check "$subsys_path" "Marginal" || return 1
+		fi
+	done
+}
+
+test_set_two_optimized_online() {
+	local IOPOLICY=$1 # "numa" "queue-depth" "round-robin"
+	shift
+	local -a RPORTS_PATHS=("$@")
+
+	# Set two optimized online
+	local numa_in_use=0
+	for subsys_path in "${RPORTS_PATHS[@]}"; do
+		if _rport_is_optimized "$subsys_path"; then
+			_rport_set_online "$subsys_path" || return 1
+			if [ "$IOPOLICY" == "numa" ] && [ $numa_in_use == 0 ]; then
+				_rport_check "$subsys_path" Online || return 1
+				numa_in_use=1
+			elif [ "$IOPOLICY" == "numa" ] && [ $numa_in_use == 1 ]; then
+				_rport_check_online "$subsys_path" Online || return 1
+				# Only one of the online paths should be in use
+				_rport_check_use "$subsys_path" Marginal || return 1
+			else
+				_rport_check "$subsys_path" Online || return 1
+			fi
+		else
+			_rport_check "$subsys_path" Marginal || return 1
+		fi
+	done
+}
+
+run_test() {
+	local IOPOLICY=$1 # "numa" "queue-depth" "round-robin"
+	shift
+	local -a ARGS=("$@")
+	local RPORTS_CNT="$(( $# / 2 ))"
+	local -a RPORTS_PATHS # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -a RPORTS_HOSTS # host1 host2
+	local PATHS_POS # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local HOSTS_POS # host1 host2
+
+	# recrate the arrays
+	for (( PATHS_POS=0; PATHS_POS < RPORTS_CNT; PATHS_POS++ )); do
+		HOSTS_POS+="$(( RPORTS_CNT + PATHS_POS ))"
+		RPORTS_PATHS+=("${ARGS[$PATHS_POS]}")
+		RPORTS_HOSTS+=("${ARGS[$HOSTS_POS]}")
+	done
+
+
+	echo Changing FC links to online
+	# Initial check to see if FC is operational and set ports to online
+	test_set_all_online "$IOPOLICY" "${RPORTS_PATHS[@]}" \
+		&& echo "Set all online: pass" || echo "Set all online: fail"
+
+	test_set_one_host_marginal "${RPORTS_HOSTS[0]}" "${RPORTS_PATHS[@]}" \
+		"${RPORTS_HOSTS[@]}" && echo "One host marginal: pass" \
+		|| echo "One host marginal: fail"
+	test_set_all_marginal "${RPORTS_PATHS[@]}" \
+		&& echo "All marginal: pass" || echo "All marginal: fail"
+
+	test_set_one_non_optimized_online "${RPORTS_PATHS[@]}" \
+		&& echo "One remote non-optimized online: pass" \
+		|| echo "One remote non-optimized online: fail"
+	test_set_all_non_optimized_online "$IOPOLICY" "${RPORTS_PATHS[@]}" \
+		&& echo "Two remote non-optimized online: pass" \
+		|| echo "Two remote non-optimized online: fail"
+
+	test_set_all_non_one_optimized_online "${RPORTS_PATHS[@]}" \
+		&& echo "Two remote non-optimized, One remote optimized: pass" \
+		|| echo "Two remote non-optimized, One remote optimized: fail"
+	test_set_all_online "$IOPOLICY" "${RPORTS_PATHS[@]}" \
+		&& echo "Set all online: pass" || echo "Set all online: fail"
+
+	test_set_all_marginal "${RPORTS_PATHS[@]}" \
+		&& echo "All marginal: pass" || echo "All marginal: fail"
+
+	test_set_one_optimized_online "${RPORTS_PATHS[@]}" \
+		&& echo "One remote optimized online: pass" \
+		|| echo "One remote optimized online: fail"
+	test_set_two_optimized_online "$IOPOLICY" "${RPORTS_PATHS[@]}" \
+		&& echo "Two remote optimized online: pass" \
+		|| echo "Two remote optimized online: fail"
+	test_set_all_online "$IOPOLICY" "${RPORTS_PATHS[@]}" \
+		&& echo "All online: pass" || echo "All online: fail"
+}
+
+_nvmet_get_rport() {
+	local PORT="$1"
+
+	local dev
+	for dev in /sys/class/nvme/nvme*; do
+		grep --quiet "io" "$dev/cntrltype" || continue
+		[ -e "$dev" ] || continue
+		dev="$(basename "$dev")"
+		grep --quiet traddr="$(_fc_traddr "$PORT")" "/sys/class/nvme/$dev/address" && echo "$dev" || true
+	done
+	# nvme9
+}
+
+run_tests() {
+	local SUBSYS_PATH="$1"
+	shift
+	local -a PORTS=("$@")
+	local -a RPORTS_PATHS # /sys/devices/virtual/nvme-subsystem/nvme-subsys3/nvme9
+	local -a RPORTS_HOSTS # host1 host2
+	local rport
+
+	for port in "${PORTS[@]}"; do
+		RPORTS_HOSTS+=("$(_get_fc_host_port "$port")")
+		rport="$(_nvmet_get_rport "$port")"
+		if [ -z "$rport" ]; then
+			echo "Could not find rport for port $port"
+			return 1
+		fi
+		if [[ "$( echo "$rport" | sed -n '$=' )" -gt 1 ]]; then
+			# One traddr has multiple /sys/class/nvme/nvme devices
+			echo "Port $port has multiple rports with address"
+			grep traddr="$(_fc_traddr "$port")" /sys/class/nvme/nvme*/address
+			return 1
+		fi
+		RPORTS_PATHS+=( "${SUBSYS_PATH}/$rport")
+	done
+
+	local IOPOLICYS="numa queue-depth round-robin"
+	for IOPOLICY in $IOPOLICYS; do
+		_rport_set_iopolicy "$SUBSYS_PATH" "$IOPOLICY"
+		echo "Testing iopolicy: $IOPOLICY"
+		run_test "$IOPOLICY" "${RPORTS_PATHS[@]}" "${RPORTS_HOSTS[@]}"
+	done
+}
+
+_find_nvme_subsys() {
+	local subsys=$1
+	local subsysnqn
+	local subsys_path
+	for subsys_path in /sys/class/nvme-subsystem/nvme-subsys*; do
+		[ -e "$subsys_path" ] || continue
+		subsysnqn="$(cat "${subsys_path}/subsysnqn" 2>/dev/null)"
+		if [[ "$subsysnqn" == "$subsys" ]]; then
+			echo "$subsys_path"
+			return
+		fi
+	done
+}
+
+test() {
+	local -a ports
+	local ns
+	local fio_pid
+	local -i first_host1=0
+	local -i first_host2=0
+
+	echo "Running ${TEST_NAME}"
+
+	_setup_nvmet 2
+
+	if [[ ! -w /sys/class/fcloop/ctl/set_marginal_rport ]]; then
+		SKIP_REASONS+=("fcloop does not support set_marginal_rport")
+		_nvmet_target_cleanup
+		return 1
+	fi
+
+	_nvmet_target_setup --ports 2
+	_nvmet_target_add_ports --host_port 1 --ports 2
+
+	_get_nvmet_ports "${def_subsysnqn}" ports
+
+	for port in "${ports[@]}"; do
+		_setup_nvmet_port_ana "${port}" 1 "non-optimized"
+		if [[ $(_get_fc_host_port "${port}") == 0 ]]; then
+			if [ $first_host1 -eq 1 ]; then
+				continue
+			fi
+			first_host1=1
+		else
+			if [ $first_host2 -eq 1 ]; then
+				continue
+			fi
+			first_host2=1
+		fi
+		_setup_nvmet_port_ana "${port}" 1 "optimized"
+	done
+
+	for port in "${ports[@]}"; do
+		_nvme_connect_subsys --port "${port}" --no-wait-ns || return 1
+	done
+
+	# start fio job
+	ns=$(_find_nvme_ns "$def_subsys_uuid")
+	echo "Starting background I/O"
+	_run_fio_verify_io --filename="/dev/${ns}" \
+			   --group_reporting --ramp_time=5 \
+			   --time_based --runtime=1m &> "$FULL" &
+	fio_pid=$!
+	sleep 10
+
+	run_tests "$(_find_nvme_subsys "${def_subsysnqn}")" "${ports[@]}"
+
+	# Stop background I/O
+	echo "Stopping background I/O"
+	{ kill "$fio_pid"; wait; } &> /dev/null
+
+	_nvme_disconnect_subsys
+	_nvmet_target_cleanup
+
+	echo "Test complete"
+}
diff --git a/tests/nvme/070.out b/tests/nvme/070.out
new file mode 100644
index 0000000..61d5173
--- /dev/null
+++ b/tests/nvme/070.out
@@ -0,0 +1,43 @@
+Running nvme/070
+Starting background I/O
+Testing iopolicy: numa
+Changing FC links to online
+Set all online: pass
+One host marginal: pass
+All marginal: pass
+One remote non-optimized online: pass
+Two remote non-optimized online: pass
+Two remote non-optimized, One remote optimized: pass
+Set all online: pass
+All marginal: pass
+One remote optimized online: pass
+Two remote optimized online: pass
+All online: pass
+Testing iopolicy: queue-depth
+Changing FC links to online
+Set all online: pass
+One host marginal: pass
+All marginal: pass
+One remote non-optimized online: pass
+Two remote non-optimized online: pass
+Two remote non-optimized, One remote optimized: pass
+Set all online: pass
+All marginal: pass
+One remote optimized online: pass
+Two remote optimized online: pass
+All online: pass
+Testing iopolicy: round-robin
+Changing FC links to online
+Set all online: pass
+One host marginal: pass
+All marginal: pass
+One remote non-optimized online: pass
+Two remote non-optimized online: pass
+Two remote non-optimized, One remote optimized: pass
+Set all online: pass
+All marginal: pass
+One remote optimized online: pass
+Two remote optimized online: pass
+All online: pass
+Stopping background I/O
+Test complete
-- 
2.54.0

Re: [PATCH v2 0/2] Test multipath and marginal ports

[John Meneghini]

This is a great improvement and we now have a functioning blkstest 070 to test the FPIN LI kernel patches.

The problem is: I don't think we are done yet.  In my own private testing and development with these patches I've been working on a next-version test that combines the ANA states from test/nvme/057 with test/nvme/070. So I am working on a test 071.

The good news is: everything now works with test/nvme/070.  The bad news is: the kernel patches are not done.

What I've found is: a long as the ANA states are all optimized or non-optimized everything works.  However, once we throw in an inaccessible state to the mix, we run into serious problems.

At this point in development I don't care about the test failures in my test/nvme/071 script. I expect the script to bug out because it doesn't understand the inaccessible state. The test sill continues flipping rports in and out of the marginal state and keeps going. That's what it is designed to do. That means it is testing all of the code paths in the kernel patches.

The problem is: when turning marginal paths on and off with controllers that are in the inaccessible state, the path selection algorithm in the kernel fails and we end up with the following:

[Wed Aug 19 16:25:42 2026] nvme_ns_head_submit_bio: 6 callbacks suppressed
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
[Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O

At this point the fio jobs are still running but there is no progress.  This means no path was found and ALL of the IOs got re-queued. And if we flip the marginal state off on all of the controllers IO continues to be hung.  The IO scheduler is hung and there is no possibility of getting it restarted again.

So this is a really serious bug in the kernel patches and we can't ship this stuff until we fix the problem.

This IO re-requeing problem should NEVER happen - no matter what the state of the marginal paths.

So I can recommend that Shinichiro test these patches with the current upstream kernel patches:

   https://lore.kernel.org/linux-nvme/20260812181300.3712426-1-jtaubepe@redhat.com/
   
But there will be another version of Kernel patches and a V3 of this patch set will be forth coming.
  
John A. Meneghini
Senior Principal Platform Storage Engineer
RHEL SST - Platform Storage Group
jmeneghi@redhat.com

On 8/19/26 16:04, Jesse Taube wrote:
> Tests for the upcoming nvme-fc: FPIN link integrity handling set.
> It tests for various multipath and marginal port
> scenarios, while confirming the port usage and state. The test is
> intended to emulate receiving an FPIN event in a multipath environment.
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=220329
> Link: https://github.com/linux-blktests/blktests/pull/264
> 
> Jesse Taube (2):
>    nvme: Add _setup_nvmet_port_marginal
>    nvme/070: Test multipath and marginal ports
> 
>   common/nvme        |  31 +++
>   tests/nvme/070     | 613 +++++++++++++++++++++++++++++++++++++++++++++
>   tests/nvme/070.out |  43 ++++
>   3 files changed, 687 insertions(+)
>   create mode 100755 tests/nvme/070
>   create mode 100644 tests/nvme/070.out
> 

Re: [PATCH v2 0/2] Test multipath and marginal ports

[John Meneghini]

Further analysis shows that this is a preexisting issue with test/nvme/057.  So I will be trying to fix that test before I combine it with test/nvme/070.

target-vm:blktests(fpin_tests10) > sudo NVMET_TRTYPES=fc ./check tests/nvme/057
nvme/057 (tr=fc) (test nvme fabrics controller ANA failover during I/O) [passed]
     runtime  25.421s  ...  25.411s
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 8210 vs 8210
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 49169 vs 49169
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 45075 vs 45075
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 16404 vs 16404
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 28693 vs 28693
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 32790 vs 32790
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 28695 vs 28695
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: transport association event: transport detected io error
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: resetting controller
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: create association : host wwpn 0x20001100aa000001  rport wwpn 0x20001100ab000003: NQN "blktests-subsystem-1"
[Thu Aug 20 10:06:04 2026] (NULL device *): {2:1} Association created
[Thu Aug 20 10:06:04 2026] nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[Thu Aug 20 10:06:04 2026] nvme nvme5: NVME-FC{3}: controller connect complete
[Thu Aug 20 10:06:04 2026] (NULL device *): {2:0} Association deleted
[Thu Aug 20 10:06:04 2026] (NULL device *): {2:0} Association freed
[Thu Aug 20 10:06:04 2026] (NULL device *): Disconnect LS failed: No Association
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 45084 vs 45084
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 16413 vs 16413
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 8222 vs 8222
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: transport association event: transport detected io error
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: resetting controller
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 45087 vs 45087
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 24608 vs 24608
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 4113 vs 4113
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 40978 vs 40978
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 53267 vs 53267
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 24596 vs 24596
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 53269 vs 53269
[Thu Aug 20 10:06:16 2026] nvme nvme2: NVME-FC{0}: io failed due to bad NVMe_ERSP: iu len 8, xfr len 4096 vs 0, status code 0, cmdid 8214 vs 8214
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O
[Thu Aug 20 10:06:16 2026] block nvme2n1: no usable path - requeuing I/O

On 8/19/26 17:49, John Meneghini wrote:
> This is a great improvement and we now have a functioning blkstest 070 to test the FPIN LI kernel patches.
> 
> The problem is: I don't think we are done yet.  In my own private testing and development with these patches I've been working on a next-version test that combines the ANA states from test/nvme/057 with test/nvme/070. So I am working on a test 071.
> 
> The good news is: everything now works with test/nvme/070.  The bad news is: the kernel patches are not done.
> 
> What I've found is: a long as the ANA states are all optimized or non-optimized everything works.  However, once we throw in an inaccessible state to the mix, we run into serious problems.
> 
> At this point in development I don't care about the test failures in my test/nvme/071 script. I expect the script to bug out because it doesn't understand the inaccessible state. The test sill continues flipping rports in and out of the marginal state and keeps going. That's what it is designed to do. That means it is testing all of the code paths in the kernel patches.
> 
> The problem is: when turning marginal paths on and off with controllers that are in the inaccessible state, the path selection algorithm in the kernel fails and we end up with the following:
> 
> [Wed Aug 19 16:25:42 2026] nvme_ns_head_submit_bio: 6 callbacks suppressed
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O
> 
> At this point the fio jobs are still running but there is no progress.  This means no path was found and ALL of the IOs got re-queued. And if we flip the marginal state off on all of the controllers IO continues to be hung.  The IO scheduler is hung and there is no possibility of getting it restarted again.
> 
> So this is a really serious bug in the kernel patches and we can't ship this stuff until we fix the problem.
> 
> This IO re-requeing problem should NEVER happen - no matter what the state of the marginal paths.
> 
> So I can recommend that Shinichiro test these patches with the current upstream kernel patches:
> 
>    https://lore.kernel.org/linux-nvme/20260812181300.3712426-1-jtaubepe@redhat.com/
> But there will be another version of Kernel patches and a V3 of this patch set will be forth coming.
> 
> John A. Meneghini
> Senior Principal Platform Storage Engineer
> RHEL SST - Platform Storage Group
> jmeneghi@redhat.com
> 
> On 8/19/26 16:04, Jesse Taube wrote:
>> Tests for the upcoming nvme-fc: FPIN link integrity handling set.
>> It tests for various multipath and marginal port
>> scenarios, while confirming the port usage and state. The test is
>> intended to emulate receiving an FPIN event in a multipath environment.
>>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=220329
>> Link: https://github.com/linux-blktests/blktests/pull/264
>>
>> Jesse Taube (2):
>>    nvme: Add _setup_nvmet_port_marginal
>>    nvme/070: Test multipath and marginal ports
>>
>>   common/nvme        |  31 +++
>>   tests/nvme/070     | 613 +++++++++++++++++++++++++++++++++++++++++++++
>>   tests/nvme/070.out |  43 ++++
>>   3 files changed, 687 insertions(+)
>>   create mode 100755 tests/nvme/070
>>   create mode 100644 tests/nvme/070.out
>>
> 

Re: [PATCH v2 0/2] Test multipath and marginal ports

[Shin'ichiro Kawasaki]

On Aug 19, 2026 / 17:49, John Meneghini wrote:
[...]
> So I can recommend that Shinichiro test these patches with the current upstream kernel patches:
> 
>   https://lore.kernel.org/linux-nvme/20260812181300.3712426-1-jtaubepe@redhat.com/
> But there will be another version of Kernel patches and a V3 of this patch set will be forth coming.

Jesse, thanks for this v2 series. The blktests side patches look good to me.
Also I ran the new test case with the kernel side patches on top of v7.2 kernel,
and the test case passed. Good. I think the last piece is the kernel side
patches. I will wait the series get settled.