[PATCH 0/4] NVMe: add support for mlx5 RDMA HW and test enhancements

[PATCH 0/4] NVMe: add support for mlx5 RDMA HW and test enhancements

[Max Gurtovoy]

Hi,

This patch series introduces several improvements to NVMe-oF support and
testing in blktests, with a focus on adding mlx5 hardware support for NVMe/RDMA
testing.

In addition, update nvme/061 test to support the new restriction added
to NVMe target driver that doesn't allow modifying the qid_max attribute
for discovered subsystems.

Max Gurtovoy (4):
  nvme/061: increase nvmf state wait timeout to 10s
  nvme: add support for mlx5 RDMA devices
  nvme/002: extend test to support TCP and RDMA transports
  nvmet: add --qid-max support for subsystem creation and setup

 common/multipath-over-rdma |  7 +++++++
 common/nvme                | 34 +++++++++++++++++++++++++++++++---
 tests/nvme/002             |  2 +-
 tests/nvme/048             |  7 ++++---
 tests/nvme/061             |  4 ++--
 5 files changed, 45 insertions(+), 9 deletions(-)

-- 
2.18.1

[PATCH 1/4] nvme/061: increase nvmf state wait timeout to 10s

[Max Gurtovoy]

Real RDMA hardware devices require more time to create resources, so
increase the wait time for subsystem state transitions from the default
5 to 10 seconds.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 tests/nvme/061 | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/nvme/061 b/tests/nvme/061
index b47e626..f2894dc 100755
--- a/tests/nvme/061
+++ b/tests/nvme/061
@@ -47,12 +47,12 @@ test() {
 
 		_nvmet_target_cleanup
 
-		_nvmf_wait_for_state "${def_subsysnqn}" "connecting" || return 1
+		_nvmf_wait_for_state "${def_subsysnqn}" "connecting" 10 || return 1
 		echo "state: $(cat "${state_file}")"
 
 		_nvmet_target_setup
 
-		_nvmf_wait_for_state "${def_subsysnqn}" "live" || return 1
+		_nvmf_wait_for_state "${def_subsysnqn}" "live" 10 || return 1
 		echo "state: $(cat "${state_file}")"
 	done
 
-- 
2.18.1

Re: [PATCH 1/4] nvme/061: increase nvmf state wait timeout to 10s

[Chaitanya Kulkarni]

On 9/28/25 04:51, Max Gurtovoy wrote:
> Real RDMA hardware devices require more time to create resources, so
> increase the wait time for subsystem state transitions from the default
> 5 to 10 seconds.
>
> Signed-off-by: Max Gurtovoy<mgurtovoy@nvidia.com>

Looks good.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>

-ck

[PATCH 2/4] nvme: add support for mlx5 RDMA devices

[Max Gurtovoy]

Until now, the tests for RDMA transport were only supported on top of
software RDMA devices (rxe/siw). This commit adds support for running
tests on top of mlx5 hardware RDMA devices, allowing for more
comprehensive testing of NVMe-oF RDMA functionality.

Prerequisites:
- RDMA network interfaces must be configured prior to running tests
- User should set USE_MLX5=1 environment variable to use mlx5 devices

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 common/multipath-over-rdma |  7 +++++++
 common/nvme                | 16 +++++++++++++---
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/common/multipath-over-rdma b/common/multipath-over-rdma
index e157e0a..9367662 100644
--- a/common/multipath-over-rdma
+++ b/common/multipath-over-rdma
@@ -98,6 +98,13 @@ rdma_network_interfaces() {
 		sort -u
 }
 
+# Lists RDMA capable network interface names associated with mlx5 devices, e.g. ib0 ib1.
+rdma_mlx5_network_interfaces() {
+	rdma link show | grep mlx5 |
+		sed -n 's/^.*[[:blank:]]netdev[[:blank:]]\+\([^[:blank:]]*\)[[:blank:]]*/\1/p' |
+		sort -u
+}
+
 # Check whether any stacked block device holds block device $1. If so, echo
 # the name of the holder.
 held_by() {
diff --git a/common/nvme b/common/nvme
index 3d43790..01d054d 100644
--- a/common/nvme
+++ b/common/nvme
@@ -218,7 +218,9 @@ _cleanup_nvmet() {
 	fi
 	modprobe -rq nvmet 2>/dev/null
 	if [[ "${nvme_trtype}" == "rdma" ]]; then
-		stop_soft_rdma
+		if [ -z "$USE_MLX5" ]; then
+			stop_soft_rdma
+		fi
 	fi
 
 	_cleanup_blkdev
@@ -243,8 +245,14 @@ _setup_nvmet() {
 	fi
 	modprobe -q nvme-"${nvme_trtype}"
 	if [[ "${nvme_trtype}" == "rdma" ]]; then
-		start_soft_rdma
-		for i in $(rdma_network_interfaces)
+		local interfaces
+		if [ -z "$USE_MLX5" ]; then
+			start_soft_rdma
+			interfaces="$(rdma_network_interfaces)"
+		else
+			interfaces="$(rdma_mlx5_network_interfaces)"
+		fi
+		for i in $interfaces
 		do
 			if [[ "${nvme_adrfam}" == "ipv6" ]]; then
 				ipv6_addr=$(get_ipv6_ll_addr "$i")
@@ -1180,6 +1188,8 @@ _nvme_requires() {
 		_have_program rdma
 		if [ -n "$USE_RXE" ]; then
 			_have_driver rdma_rxe
+		elif [ -n "$USE_MLX5" ]; then
+			_have_driver mlx5_ib
 		else
 			_have_driver siw
 		fi
-- 
2.18.1

Re: [PATCH 2/4] nvme: add support for mlx5 RDMA devices

[Chaitanya Kulkarni]

On 9/28/25 04:51, Max Gurtovoy wrote:
> Until now, the tests for RDMA transport were only supported on top of
> software RDMA devices (rxe/siw). This commit adds support for running
> tests on top of mlx5 hardware RDMA devices, allowing for more
> comprehensive testing of NVMe-oF RDMA functionality.
>
> Prerequisites:
> - RDMA network interfaces must be configured prior to running tests
> - User should set USE_MLX5=1 environment variable to use mlx5 devices
>
> Signed-off-by: Max Gurtovoy<mgurtovoy@nvidia.com>

Looks good.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>

-ck

Re: [PATCH 2/4] nvme: add support for mlx5 RDMA devices

[Shinichiro Kawasaki]

On Sep 28, 2025 / 14:51, Max Gurtovoy wrote:
> Until now, the tests for RDMA transport were only supported on top of
> software RDMA devices (rxe/siw). This commit adds support for running
> tests on top of mlx5 hardware RDMA devices, allowing for more
> comprehensive testing of NVMe-oF RDMA functionality.
> 
> Prerequisites:
> - RDMA network interfaces must be configured prior to running tests
> - User should set USE_MLX5=1 environment variable to use mlx5 devices

Hello Max, it's great that now we can run blktests with mlx5 hardware :)

As to this patch, have you considered to use NVME_TARGET_CONTROL?

As the commit 5b6d89ced118 ("nvme/rc: introduce remote target support")
explains, NVME_TARGET_CONTROL was introduced to skip the generic target
setup/cleanup code to run the tests with a real target.

This patch introduced branches with the USE_MLX5 variable. If we take this
approach, we will need to add more branches when we support other hardware in
future. If it is possible, I guess NVME_TARGET_CONTROL will be cleaner.

As to the script to be set as NVME_TARGET_CONTROL, we can find an example:
contrib/nvme_target_control.py. But it might be too complicated for your use
case. I'm guessing the script for your use case will be as simple as this patch,
and it might be worth adding to contrib/.

[PATCH 3/4] nvme/002: extend test to support TCP and RDMA transports

[Max Gurtovoy]

Expand the transport support for the discovery test from loop-only
to include TCP and RDMA transports. This allows testing NVMe-oF
discovery functionality across different transport protocols.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 tests/nvme/002 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/nvme/002 b/tests/nvme/002
index b87570a..64c84f9 100755
--- a/tests/nvme/002
+++ b/tests/nvme/002
@@ -12,7 +12,7 @@ DESCRIPTION="create many subsystems and test discovery"
 requires() {
 	_nvme_requires
 	_have_loop
-	_require_nvme_trtype_is_loop
+	_require_nvme_trtype loop tcp rdma
 }
 
 set_conditions() {
-- 
2.18.1

Re: [PATCH 3/4] nvme/002: extend test to support TCP and RDMA transports

[Chaitanya Kulkarni]

On 9/28/25 04:51, Max Gurtovoy wrote:
> Expand the transport support for the discovery test from loop-only
> to include TCP and RDMA transports. This allows testing NVMe-oF
> discovery functionality across different transport protocols.
>
> Signed-off-by: Max Gurtovoy<mgurtovoy@nvidia.com>

Looks good.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>

-ck

[PATCH 4/4] nvmet: add --qid-max support for subsystem creation and setup

[Max Gurtovoy]

Introduce support for the --qid-max argument to both
_create_nvmet_subsystem and _nvmet_target_setup functions, allowing
configuration of the maximum queue ID through the "attr_qid_max" sysfs
attribute when available. This enables more flexible queue management
for NVMe subsystems.

Additionally, update test 048 to set qid_max via the new --qid-max
workflow. This will follow the new logic added to the NVMe target
driver, which forbid changing the attr_qid_max for a discovered
subsystem.

Also increase the sleep durations to adjust real RDMA HW devices.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 common/nvme    | 18 ++++++++++++++++++
 tests/nvme/048 |  7 ++++---
 2 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/common/nvme b/common/nvme
index 01d054d..3a883ff 100644
--- a/common/nvme
+++ b/common/nvme
@@ -752,6 +752,7 @@ _create_nvmet_subsystem() {
 	local blkdev
 	local uuid="${def_subsys_uuid}"
 	local resv_enable
+	local qid_max
 	local cfs_path
 	local -a ARGS
 
@@ -773,6 +774,10 @@ _create_nvmet_subsystem() {
 				resv_enable="--resv_enable";
 				shift 1
 				;;
+			--qid-max)
+				qid_max="$2"
+				shift 2
+				;;
 			*)
 				echo "WARNING: unknown argument: $1"
 				shift
@@ -794,6 +799,11 @@ _create_nvmet_subsystem() {
 	if [[ -n "$resv_enable" ]]; then
 		ARGS+=("${resv_enable}")
 	fi
+	if [[ -f "${cfs_path}/attr_qid_max" ]]; then
+		if [[ -n "$qid_max" ]]; then
+			echo "${qid_max}" > "${cfs_path}/attr_qid_max"
+		fi
+	fi
 	_create_nvmet_ns "${ARGS[@]}" > /dev/null
 }
 
@@ -971,6 +981,7 @@ _nvmet_target_setup() {
 	local port p
 	local resv_enable=""
 	local num_ports=1
+	local qid_max
 	local tls="none"
 	local -a ARGS
 
@@ -1012,6 +1023,10 @@ _nvmet_target_setup() {
 				tls="required"
 				shift 1
 				;;
+			--qid-max)
+				qid_max="$2"
+				shift 2
+				;;
 			*)
 				echo "WARNING: unknown argument: $1"
 				shift
@@ -1054,6 +1069,9 @@ _nvmet_target_setup() {
 	if [[ -n "${resv_enable}" ]]; then
 		ARGS+=("${resv_enable}")
 	fi
+	if [[ -n "${qid_max}" ]]; then
+		ARGS+=(--qid-max "${qid_max}")
+	fi
 	_create_nvmet_subsystem "${ARGS[@]}"
 
 	p=0
diff --git a/tests/nvme/048 b/tests/nvme/048
index afd9272..c6f4d9f 100755
--- a/tests/nvme/048
+++ b/tests/nvme/048
@@ -37,7 +37,7 @@ nvmf_check_queue_count() {
 			return 1
 		fi
 
-		sleep 1
+		sleep 2
 
 		retries=$((retries - 1))
 		queue_count_file=$(cat /sys/class/nvme-fabrics/ctl/"${nvmedev}"/queue_count)
@@ -58,9 +58,10 @@ set_qid_max() {
 	local subsys_name="$1"
 	local qid_max="$2"
 
-	set_nvmet_attr_qid_max "${subsys_name}" "${qid_max}"
+	_nvmet_target_cleanup
+	_nvmet_target_setup --blkdev file --qid-max "${qid_max}"
 	nvmf_check_queue_count "${subsys_name}" "${qid_max}" || return 1
-	_nvmf_wait_for_state "${subsys_name}" "live" || return 1
+	_nvmf_wait_for_state "${subsys_name}" "live" 10 || return 1
 
 	return 0
 }
-- 
2.18.1

Re: [PATCH 4/4] nvmet: add --qid-max support for subsystem creation and setup

[Chaitanya Kulkarni]

On 9/28/25 04:51, Max Gurtovoy wrote:
> Introduce support for the --qid-max argument to both
> _create_nvmet_subsystem and _nvmet_target_setup functions, allowing
> configuration of the maximum queue ID through the "attr_qid_max" sysfs
> attribute when available. This enables more flexible queue management
> for NVMe subsystems.
>
> Additionally, update test 048 to set qid_max via the new --qid-max
> workflow. This will follow the new logic added to the NVMe target
> driver, which forbid changing the attr_qid_max for a discovered
> subsystem.
>
> Also increase the sleep durations to adjust real RDMA HW devices.
>
> Signed-off-by: Max Gurtovoy<mgurtovoy@nvidia.com>
> ---

Looks good.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>

-ck

Re: [PATCH 4/4] nvmet: add --qid-max support for subsystem creation and setup

[Shinichiro Kawasaki]

On Sep 28, 2025 / 14:51, Max Gurtovoy wrote:
> Introduce support for the --qid-max argument to both
> _create_nvmet_subsystem and _nvmet_target_setup functions, allowing
> configuration of the maximum queue ID through the "attr_qid_max" sysfs
> attribute when available. This enables more flexible queue management
> for NVMe subsystems.
> 
> Additionally, update test 048 to set qid_max via the new --qid-max
> workflow. This will follow the new logic added to the NVMe target
> driver, which forbid changing the attr_qid_max for a discovered
> subsystem.
> 
> Also increase the sleep durations to adjust real RDMA HW devices.
> 
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
[...]
> diff --git a/tests/nvme/048 b/tests/nvme/048
> index afd9272..c6f4d9f 100755
> --- a/tests/nvme/048
> +++ b/tests/nvme/048
> @@ -37,7 +37,7 @@ nvmf_check_queue_count() {
>  			return 1
>  		fi
>  
> -		sleep 1
> +		sleep 2
>  
>  		retries=$((retries - 1))
>  		queue_count_file=$(cat /sys/class/nvme-fabrics/ctl/"${nvmedev}"/queue_count)
> @@ -58,9 +58,10 @@ set_qid_max() {
>  	local subsys_name="$1"
>  	local qid_max="$2"
>  
> -	set_nvmet_attr_qid_max "${subsys_name}" "${qid_max}"

The call to set_nvmet_attr_qid_max() is removed here, but the function is still
left in nvme/048. I suggest to remove it.

> +	_nvmet_target_cleanup
> +	_nvmet_target_setup --blkdev file --qid-max "${qid_max}"
>  	nvmf_check_queue_count "${subsys_name}" "${qid_max}" || return 1
> -	_nvmf_wait_for_state "${subsys_name}" "live" || return 1
> +	_nvmf_wait_for_state "${subsys_name}" "live" 10 || return 1

Re: [PATCH 0/4] NVMe: add support for mlx5 RDMA HW and test enhancements

[Shinichiro Kawasaki]

Max, thanks for the series. I made two comments on the patches. One more nit
comment is about the patch subject: please add the prefix 'blktests' to clarify
that the patches are for blktests, like "[PATCH blktests]". Thanks!